git: dfregress - An automated test driver and framework
Alex Hornung
alexh at crater.dragonflybsd.org
Mon Nov 14 10:36:54 PST 2011
commit 6a7006f10516182dbc2d134ecce752158f3ce61b
Author: Alex Hornung <ahornung at gmail.com>
Date: Sun Oct 30 01:21:59 2011 +0000
dfregress - An automated test driver and framework
* dfregress is a simple test framework and test automation driver. It
supports both normal userland testcases and kernel test cases.
* It's key aim is to make it simple, if not outright dead easy, to
write test cases. A test case is a simple program using
printf/fprintf and exit - no magic needed. A kernel test case is a
very small module that just needs to implement a few functions and
call a logging and result function.
* Sample output of the text frontend (Frontends are explained further
down the text): http://leaf.dragonflybsd.org/~alexh/dfregress.txt
* dfregress is very UNIXy, it uses makefiles to build the testcases,
stdout/stderr redirection from the testcases (no fancy output
functions needed in the testcases) and evaluates the normal return
value of the testcase (no need to call fancy functions).
* For kernel testcases it is a bit different - you do have to call
functions to log output and to log the result, but it is very simple,
hardly any overhead.
* The test driver assumes that testcases are in the testcases/
directory, but it supports several command line options.
* The tests to run, including several options, are specified in the
runlist file. An example runlist including all the testcases is
included as config/runlist.run. Options that can be specified are:
- timeout: (in seconds) after which the test is aborted if it hasn't
finished.
- test type (userland, kernel or buildonly)
- make: which 'make' tool to use to build the test cases. Defaults to
'make', but could also be 'gmake', etc.
- nobuild: doesn't build the testcase and tries to directly execute
it.
- pre, post: external pre-run and post-run commands, e.g. to set up a
vn device and to tear it down. This is to avoid duplication in test
cases, the common setup can be factored out.
- intpre, intpost: similar to the above, but it assumes that the
testcase, when passed the parameter 'pre', will do the pre-setup,
and when passed 'post' will do the post-test cleanup/setup/etc.
- any number of command line arguments that are passed to the test
case. (See the crypto/ test examples in the runlist).
* A range of sample testcases are available in
test/dfregress/testcases/sample, including a kernel testcase sample.
* Note that many of the test cases in the testcases/ directory have
been copied from elsewhere in the main repository and are,
temporarily at least, duplicated.
* The test driver is completely separated from the frontends. The test
driver outputs the test run results in an XML-like format (plist)
that can easily be parsed using proplib. Python and Ruby also have
plist modules that could be used to parse the output.
* The only available frontend is a simple C program that will parse the
intermediate format to an easy to read plain text format. Additional
frontends can be written in any language, as long as it is possible
to load plists. Frontends are in the fe/ directory.
* XXX: the default options (currently just the timeout) are still
hardcoded in config.c.
* The NOTES file gives details on how the test execution occurs and
what result code can be raised at which point. This document, and a look
at the generated plist, is all you need to write a new frontend that,
for example, generates beautiful HTML output.
For completeness sake, a part of NOTES is reproduced under the main commit
message - specifically the part detailing the execution of a single test
case.
======
Execution of a single test case:
======
1) chdir to testcase directory
- if it fails, set RESULT_PREFAIL (sysbuf is of interest), goto (6)
2) build testcase (make) (unless nobuild flag is set).
+ build_buf is used for stdout/stderr
- if there is an internal driver error (that leads to not running the
build command), set RESULT_PREFAIL (sysbuf is of interest), goto (6)
- if the build command has a non-zero exit value, set the result to
BUILDFAIL, unless it's a buildonly test case, in which it is set to
the actual result value (TIMEOUT, SIGNALLED, FAIL)
goto (6)
3) run 'pre' command if intpre or pre is set.
+ precmd_buf is used for stdout/stderr
- if there is an internal driver error (that leads to not running the
command), set RESULT_PREFAIL (sysbuf is of interest), goto (6)
- if the pre command has a non-zero exit value, set RESULT_PREFAIL and
goto (6)
4) run actual testcase, depending on type
+ stdout_buf is used for stdout
+ stderr_buf is used for stderr
- for BUILDONLY: set RESULT_PASS since the build already succeeded
- for userland and kernel: run the testcase, possibly as a different
user (depending on the runas option), set the result to the actual
result value (TIMEOUT, SIGNALLED, FAIL, NOTRUN)
- if there is an internal driver error (that leads to not running the
command), RESULT_NOTRUN is set (sysbuf is of interest)
5) run 'post' command if intpost or post is set.
+ postcmd_buf is used for stdout/stderr
- if there is an internal driver error (that leads to not running the
command), set RESULT_POSTFAIL (sysbuf is of interest), goto (6)
- if the post command has a non-zero exit value, set RESULT_POSTFAIL
and goto (6)
6) clean testcase directory (make clean) (unless nobuild flag is set).
+ cleanup_buf is used for stdout/stderr and system (driver error) buffer
- no further action.
7) results are saved.
Summary of changes:
test/dfregress/NOTES | 76 +++
test/dfregress/config/defaults.conf | 4 +
test/dfregress/config/runlist.run | 53 ++
test/dfregress/driver/Makefile | 13 +
test/dfregress/driver/config.c | 59 ++
test/dfregress/driver/config.h | 30 +
test/dfregress/driver/kernel.c | 145 ++++
test/dfregress/driver/kernel.h | 30 +
test/dfregress/driver/main.c | 103 +++
test/dfregress/driver/parser.c | 218 ++++++
test/dfregress/driver/parser.h | 35 +
test/dfregress/driver/runlist.c | 312 +++++++++
test/dfregress/driver/runlist.h | 40 ++
test/dfregress/driver/testcase.c | 695 ++++++++++++++++++++
test/dfregress/driver/testcase.h | 116 ++++
test/dfregress/driver/userland.c | 295 +++++++++
test/dfregress/driver/userland.h | 34 +
test/dfregress/fe/text/Makefile | 18 +
test/dfregress/fe/text/fe_text.c | 223 +++++++
test/dfregress/framework/dfregress.h | 11 +
test/dfregress/framework/tbridge.h | 72 ++
test/dfregress/framework/test.txt | 2 +
test/dfregress/kernel_bridge/Makefile | 4 +
test/dfregress/kernel_bridge/dfregress_bridge.c | 344 ++++++++++
test/dfregress/kernel_bridge/safe_mem.c | 191 ++++++
test/dfregress/testcases/Makefile | 10 +
test/dfregress/testcases/compiler/Makefile | 3 +
test/dfregress/testcases/compiler/div128/Makefile | 4 +
test/dfregress/testcases/compiler/div128/div128.c | 12 +
test/dfregress/testcases/crypto/Makefile | 3 +
test/{ => dfregress/testcases}/crypto/aes/Makefile | 0
.../{ => dfregress/testcases}/crypto/aes/aestest.c | 0
.../testcases}/crypto/aes/ecbnk44.txt | 0
.../testcases}/crypto/aes/ecbnk48.txt | 0
.../testcases}/crypto/aes/ecbnt44.txt | 0
.../testcases}/crypto/aes/ecbnt48.txt | 0
.../testcases}/crypto/aes/ecbvk44.txt | 0
.../testcases}/crypto/aes/ecbvk48.txt | 0
.../testcases}/crypto/aes/ecbvt44.txt | 0
.../testcases}/crypto/aes/ecbvt48.txt | 0
.../testcases}/crypto/aesctr/Makefile | 0
.../testcases}/crypto/aesctr/aesctr.c | 0
.../testcases}/crypto/aesxts/Makefile | 0
.../testcases}/crypto/aesxts/aes_xts.c | 0
.../testcases}/crypto/serpent/Makefile | 0
.../testcases}/crypto/serpent/serpent_test.c | 0
.../testcases}/crypto/serpent/serpentecb_vk.txt | 0
.../testcases}/crypto/serpent/serpentecb_vt.txt | 0
.../testcases}/crypto/twofish/Makefile | 0
.../testcases}/crypto/twofish/twofish_test.c | 0
.../testcases}/crypto/twofish/twofishecb_vk.txt | 0
.../testcases}/crypto/twofish/twofishecb_vt.txt | 0
test/dfregress/testcases/io/Makefile | 6 +
test/dfregress/testcases/io/kqueue_1/Makefile | 4 +
test/dfregress/testcases/io/kqueue_1/kqueue_1.c | 49 ++
test/dfregress/testcases/io/kqueue_2/Makefile | 4 +
.../dfregress/testcases/io/kqueue_2/kqueue_2.c | 0
test/dfregress/testcases/io/pselect_1/Makefile | 4 +
test/dfregress/testcases/io/pselect_1/pselect_1.c | 145 ++++
test/dfregress/testcases/io/select_1/Makefile | 4 +
test/dfregress/testcases/io/select_1/select_1.c | 125 ++++
test/dfregress/testcases/io/select_2/Makefile | 4 +
.../dfregress/testcases/io/select_2/select_2.c | 0
test/dfregress/testcases/io/select_3/Makefile | 4 +
.../dfregress/testcases/io/select_3/select_3.c | 0
test/dfregress/testcases/io/select_4/Makefile | 4 +
.../dfregress/testcases/io/select_4/select_4.c | 0
test/dfregress/testcases/io/sendfd_1/Makefile | 4 +
test/dfregress/testcases/io/sendfd_1/sendfd_1.c | 93 +++
test/dfregress/testcases/mem/Makefile | 3 +
test/dfregress/testcases/mem/mmap_1/Makefile | 5 +
test/dfregress/testcases/mem/mmap_1/mmap_1.c | 48 ++
.../testcases/mem/mmap_madvise_1/Makefile | 4 +
.../testcases/mem/mmap_madvise_1/mmap_madvise_1.c | 97 +++
test/dfregress/testcases/misc/Makefile | 3 +
test/dfregress/testcases/misc/sh_1/Makefile | 4 +
test/dfregress/testcases/misc/sh_1/sh_1.sh | 11 +
test/dfregress/testcases/priv/Makefile | 3 +
test/dfregress/testcases/priv/setreuid/Makefile | 4 +
test/dfregress/testcases/priv/setreuid/setreuid.c | 38 ++
test/dfregress/testcases/sample/Makefile | 5 +
test/dfregress/testcases/sample/test1/Makefile | 4 +
test/dfregress/testcases/sample/test1/test1.c | 15 +
test/dfregress/testcases/sample/test2/Makefile | 4 +
test/dfregress/testcases/sample/test2/test2.c | 13 +
test/dfregress/testcases/sample/test3/Makefile | 4 +
test/dfregress/testcases/sample/test3/test3.c | 9 +
test/dfregress/testcases/sample/test4/Makefile | 4 +
test/dfregress/testcases/sample/test4/test4.c | 20 +
test/dfregress/testcases/sample/test5/Makefile | 4 +
test/dfregress/testcases/sample/test5/test5.c | 9 +
test/dfregress/testcases/sample/testb1/Makefile | 4 +
test/dfregress/testcases/sample/testb1/testb1.c | 7 +
test/dfregress/testcases/sample/testk1/Makefile | 4 +
test/dfregress/testcases/sample/testk1/testk1.c | 25 +
test/dfregress/testcases/sample/testk2/Makefile | 4 +
test/dfregress/testcases/sample/testk2/testk1.c | 25 +
test/dfregress/testcases/threads/Makefile | 3 +
.../testcases/threads/umtx_errno/Makefile | 5 +
.../testcases/threads/umtx_errno/umtx_errno.c | 95 +++
100 files changed, 4088 insertions(+), 0 deletions(-)
create mode 100644 test/dfregress/NOTES
create mode 100644 test/dfregress/config/defaults.conf
create mode 100644 test/dfregress/config/runlist.run
create mode 100644 test/dfregress/driver/Makefile
create mode 100644 test/dfregress/driver/config.c
create mode 100644 test/dfregress/driver/config.h
create mode 100644 test/dfregress/driver/kernel.c
create mode 100644 test/dfregress/driver/kernel.h
create mode 100644 test/dfregress/driver/main.c
create mode 100644 test/dfregress/driver/parser.c
create mode 100644 test/dfregress/driver/parser.h
create mode 100644 test/dfregress/driver/runlist.c
create mode 100644 test/dfregress/driver/runlist.h
create mode 100644 test/dfregress/driver/testcase.c
create mode 100644 test/dfregress/driver/testcase.h
create mode 100644 test/dfregress/driver/userland.c
create mode 100644 test/dfregress/driver/userland.h
create mode 100644 test/dfregress/fe/text/Makefile
create mode 100644 test/dfregress/fe/text/fe_text.c
create mode 100644 test/dfregress/framework/dfregress.h
create mode 100644 test/dfregress/framework/tbridge.h
create mode 100644 test/dfregress/framework/test.txt
create mode 100644 test/dfregress/kernel_bridge/Makefile
create mode 100644 test/dfregress/kernel_bridge/dfregress_bridge.c
create mode 100644 test/dfregress/kernel_bridge/safe_mem.c
create mode 100644 test/dfregress/testcases/Makefile
create mode 100644 test/dfregress/testcases/compiler/Makefile
create mode 100644 test/dfregress/testcases/compiler/div128/Makefile
create mode 100644 test/dfregress/testcases/compiler/div128/div128.c
create mode 100644 test/dfregress/testcases/crypto/Makefile
copy test/{ => dfregress/testcases}/crypto/aes/Makefile (100%)
copy test/{ => dfregress/testcases}/crypto/aes/aestest.c (100%)
copy test/{ => dfregress/testcases}/crypto/aes/ecbnk44.txt (100%)
copy test/{ => dfregress/testcases}/crypto/aes/ecbnk48.txt (100%)
copy test/{ => dfregress/testcases}/crypto/aes/ecbnt44.txt (100%)
copy test/{ => dfregress/testcases}/crypto/aes/ecbnt48.txt (100%)
copy test/{ => dfregress/testcases}/crypto/aes/ecbvk44.txt (100%)
copy test/{ => dfregress/testcases}/crypto/aes/ecbvk48.txt (100%)
copy test/{ => dfregress/testcases}/crypto/aes/ecbvt44.txt (100%)
copy test/{ => dfregress/testcases}/crypto/aes/ecbvt48.txt (100%)
copy test/{ => dfregress/testcases}/crypto/aesctr/Makefile (100%)
copy test/{ => dfregress/testcases}/crypto/aesctr/aesctr.c (100%)
copy test/{ => dfregress/testcases}/crypto/aesxts/Makefile (100%)
copy test/{ => dfregress/testcases}/crypto/aesxts/aes_xts.c (100%)
copy test/{ => dfregress/testcases}/crypto/serpent/Makefile (100%)
copy test/{ => dfregress/testcases}/crypto/serpent/serpent_test.c (100%)
copy test/{ => dfregress/testcases}/crypto/serpent/serpentecb_vk.txt (100%)
copy test/{ => dfregress/testcases}/crypto/serpent/serpentecb_vt.txt (100%)
copy test/{ => dfregress/testcases}/crypto/twofish/Makefile (100%)
copy test/{ => dfregress/testcases}/crypto/twofish/twofish_test.c (100%)
copy test/{ => dfregress/testcases}/crypto/twofish/twofishecb_vk.txt (100%)
copy test/{ => dfregress/testcases}/crypto/twofish/twofishecb_vt.txt (100%)
create mode 100644 test/dfregress/testcases/io/Makefile
create mode 100644 test/dfregress/testcases/io/kqueue_1/Makefile
create mode 100644 test/dfregress/testcases/io/kqueue_1/kqueue_1.c
create mode 100644 test/dfregress/testcases/io/kqueue_2/Makefile
copy tools/test/kqueue/kqueue_oob.c => test/dfregress/testcases/io/kqueue_2/kqueue_2.c (100%)
create mode 100644 test/dfregress/testcases/io/pselect_1/Makefile
create mode 100644 test/dfregress/testcases/io/pselect_1/pselect_1.c
create mode 100644 test/dfregress/testcases/io/select_1/Makefile
create mode 100644 test/dfregress/testcases/io/select_1/select_1.c
create mode 100644 test/dfregress/testcases/io/select_2/Makefile
copy tools/test/select/select_eof.c => test/dfregress/testcases/io/select_2/select_2.c (100%)
create mode 100644 test/dfregress/testcases/io/select_3/Makefile
copy tools/test/select/select_many_write.c => test/dfregress/testcases/io/select_3/select_3.c (100%)
create mode 100644 test/dfregress/testcases/io/select_4/Makefile
copy tools/test/select/select_oob.c => test/dfregress/testcases/io/select_4/select_4.c (100%)
create mode 100644 test/dfregress/testcases/io/sendfd_1/Makefile
create mode 100644 test/dfregress/testcases/io/sendfd_1/sendfd_1.c
create mode 100644 test/dfregress/testcases/mem/Makefile
create mode 100644 test/dfregress/testcases/mem/mmap_1/Makefile
create mode 100644 test/dfregress/testcases/mem/mmap_1/mmap_1.c
create mode 100644 test/dfregress/testcases/mem/mmap_madvise_1/Makefile
create mode 100644 test/dfregress/testcases/mem/mmap_madvise_1/mmap_madvise_1.c
create mode 100644 test/dfregress/testcases/misc/Makefile
create mode 100644 test/dfregress/testcases/misc/sh_1/Makefile
create mode 100644 test/dfregress/testcases/misc/sh_1/sh_1.sh
create mode 100644 test/dfregress/testcases/priv/Makefile
create mode 100644 test/dfregress/testcases/priv/setreuid/Makefile
create mode 100644 test/dfregress/testcases/priv/setreuid/setreuid.c
create mode 100644 test/dfregress/testcases/sample/Makefile
create mode 100644 test/dfregress/testcases/sample/test1/Makefile
create mode 100644 test/dfregress/testcases/sample/test1/test1.c
create mode 100644 test/dfregress/testcases/sample/test2/Makefile
create mode 100644 test/dfregress/testcases/sample/test2/test2.c
create mode 100644 test/dfregress/testcases/sample/test3/Makefile
create mode 100644 test/dfregress/testcases/sample/test3/test3.c
create mode 100644 test/dfregress/testcases/sample/test4/Makefile
create mode 100644 test/dfregress/testcases/sample/test4/test4.c
create mode 100644 test/dfregress/testcases/sample/test5/Makefile
create mode 100644 test/dfregress/testcases/sample/test5/test5.c
create mode 100644 test/dfregress/testcases/sample/testb1/Makefile
create mode 100644 test/dfregress/testcases/sample/testb1/testb1.c
create mode 100644 test/dfregress/testcases/sample/testk1/Makefile
create mode 100644 test/dfregress/testcases/sample/testk1/testk1.c
create mode 100644 test/dfregress/testcases/sample/testk2/Makefile
create mode 100644 test/dfregress/testcases/sample/testk2/testk1.c
create mode 100644 test/dfregress/testcases/threads/Makefile
create mode 100644 test/dfregress/testcases/threads/umtx_errno/Makefile
create mode 100644 test/dfregress/testcases/threads/umtx_errno/umtx_errno.c
http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/6a7006f10516182dbc2d134ecce752158f3ce61b
--
DragonFly BSD source repository
More information about the Commits
mailing list