README revision 1.2
1 1.2 apb $NetBSD: README,v 1.2 2009/09/14 17:15:32 apb Exp $ 2 1.1 martin 3 1.2 apb NOTE: New tests should use the ATF framework; see atf(7) 4 1.2 apb and the src/tests directory. 5 1.2 apb 6 1.2 apb ------------------------------------------------------------------------ 7 1.1 martin This part of the source tree contains regression tests. There are special 8 1.1 martin make targets and rules to follow. Most of these, however, are currently not 9 1.1 martin enforced, and most tests available are not conforming. 10 1.1 martin 11 1.1 martin We hope to fix this someday. If you add new tests, please try to be conforming. 12 1.1 martin 13 1.1 martin What is a regression test? 14 1.1 martin 15 1.1 martin A regression test is run by a makefile in a test directory (see below). 16 1.1 martin Each makefile may run multiple tests. 17 1.1 martin 18 1.1 martin What is a test directory? 19 1.1 martin 20 1.1 martin A directory in this part of the tree is a regression test directory. It 21 1.1 martin contains a Makefile which implements the additional "regress" target, 22 1.1 martin and runs all it's regression tests during this target. 23 1.1 martin 24 1.1 martin May the make progress be stopped on failures? 25 1.1 martin 26 1.1 martin No, the "make regress" target should succeed, unless some regression 27 1.1 martin binaries could not be build, disk is full or other catastrophic failures 28 1.1 martin outside of the tested subsystem happen. A failing regression test should 29 1.1 martin log the failure (see below), but not make the target itself fail. 30 1.1 martin 31 1.1 martin What are the possible results of a regression test? 32 1.1 martin 33 1.1 martin A test may either 34 1.1 martin 35 1.1 martin - succeed, in which case it logs "PASSED" (see below for logging details) 36 1.1 martin - fail, in which case it logs "FAILED" 37 1.1 martin - not be able to run, in which case it logs "SKIPPED" and the reason 38 1.1 martin for the skip in the comment field (see below) 39 1.1 martin 40 1.1 martin Typical reasons for tests to not being run are missing kernel options, 41 1.1 martin or missing privileges (test needs root, "make regress" is invoked by 42 1.1 martin mere mortal or vice versa). A test may not fail because of such 43 1.1 martin environmental issues, it must detect and properly log the problem. 44 1.1 martin 45 1.1 martin If a test directory contains tests that may be skipped, it should have 46 1.1 martin a README file explaining the prerequisites (e.g. needed kernel options) 47 1.1 martin 48 1.1 martin In future, we will mark affected makefiles and optimize run/skipped test 49 1.1 martin during repeated runs with differing privileges - but currently there is no 50 1.1 martin make framework in place to handle this. 51 1.1 martin 52 1.1 martin How and when does a test log results? 53 1.1 martin 54 1.1 martin If the make/environment variable ${REGRESS_LOG} is defined, the final 55 1.1 martin results (and only those) should be logged to the file named by that 56 1.1 martin variable. We will, in the future, add make targets for this purpose. 57 1.1 martin The log format is line oriented, one line used per test. Each line 58 1.1 martin consists of the directory where the Makefile lives, followed by the 59 1.1 martin test name and the result (see above: PASSED, FAILED, SKIPPED). 60 1.1 martin Following this an optional comment may be added. For SKIPPED tests the 61 1.1 martin comment is not optional. Fields are separated by spaces. 62
Indexes created Thu Oct 02 10:09:58 GMT 2025