xref: /src/external/bsd/atf/dist/NEWS

   Major changes in ATF releases

   By Julio Merino, The NetBSD Foundation

                                    Contents

    1. Changes in version 0.9

    2. Changes in version 0.8

    3. Changes in version 0.7

    4. Changes in version 0.6

    5. Changes in version 0.5

    6. Changes in version 0.4

    7. Changes in version 0.3

    8. Changes in version 0.2

    9. Changes in version 0.1

                             Changes in version 0.9

   Experimental version released on June 3rd, 2010.

     * Added atf-sh, an interpreter to process test programs written using
       the shell API. This is not really a shell interpreter by itself
       though: it is just a wrapper around the system shell that eases the
       loading of the necessary ATF libraries.

     * Removed atf-compile in favour of atf-sh.

     * Added the use.fs metadata property to test case, which is used to
       specify which test cases require file system access. This is to
       highlight dependencies on external resources more clearly and to speed
       up the execution of test suites by skipping the creation of many
       unnecessary work directories.

     * Fixed test programs to get a sane default value for their source
       directory. This means that it should not be necessary any more to pass
       -s when running test programs that do not live in the current
       directory.

     * Defining test case headers became optional. This is trivial to achieve
       in shell-based tests but a bit ugly in C and C++. In C, use the new
       ATF_TC_WITHOUT_HEAD macro to define the test case, and in C++ use
       ATF_TEST_CASE_WITHOUT_HEAD.

                             Changes in version 0.8

   Experimental version released on May 7th, 2010.

     * Test programs no longer run several test cases in a row. The execution
       of a test program now requires a test case name, and that single test
       case is executed. To execute several test cases, use the atf-run
       utility as usual.

     * Test programs no longer fork a subprocess to isolate the execution of
       test cases. They run the test case code in-process, and a crash of the
       test case will result in a crash of the test program. This is to ease
       debugging of faulty test cases.

     * Test programs no longer isolate their test cases. This means that they
       will not create temporary directories nor sanitize the environment any
       more. Yes: running a test case that depends on system state by hand
       will most likely yield different results depending on where (machine,
       directory, user environment, etc.) it is run. Isolation has been moved
       to atf-run.

     * Test programs no longer print a cryptic format (application/X-atf-tcs)
       on a special file channel. They can now print whatever they want on
       the screen. Because test programs can now only run one test case every
       time, providing controlled output is not necessary any more.

     * Test programs no longer write their status into a special file
       descriptor. Instead, they create a file with the results, which is
       later parsed by atf-run. This changes the semantics of the -r flag.

     * atf-run has been adjusted to perform the test case isolation. As a
       result, there is now a single canonical place that implements the
       isolation of test caes. In previous releases, the three language
       bindings (C, C++ and shell) had to be kept in sync with each other
       (read: not a nice thing to do at all). As a side effect of this
       change, writing bindings for other languages will be much, much easier
       from now on.

     * atf-run forks test programs on a test case basis, instead of on a test
       program basis as it did before. This is to provide the test case
       isolation that was before implemented by the test programs themselves.

     * Removed the atf-exec tool. This was used to implement test case
       isolation in atf-sh, but it is now unnecessary.

     * It is now optional to define the descr meta-data property. It has been
       proven to be mostly useless, because test cases often carry a
       descriptive name of their own.

                             Changes in version 0.7

   Experimental version released on December 22nd, 2009.

     * Added build-time checks to atf-c and atf-c++. A binding for atf-sh
       will come later.

     * Migrated all build-time checks for header files to proper ATF tests.
       This demonstrates the use of the new feature described above.

     * Added an internal API for child process management.

     * Converted all plain-text distribution documents to a Docbook canonical
       version, and include pre-generated plain text and HTML copies in the
       distribution file.

     * Simplified the contents of the Makefile.am by regenerating it from a
       canonical Makefile.am.m4 source. As a side-effect, some dependency
       specifications were fixed.

     * Migrated all checks from the check target to installcheck, as these
       require ATF to be installed.

     * Fixed sign comparison mismatches triggered by the now-enabled
       -Wsign-compare.

     * Fixed many memory and object leaks.

                             Changes in version 0.6

   Experimental version released on January 18th, 2009.

     * Make atf-exec be able to kill its child process after a certain period
       of time; this is controlled through the new -t option.

     * Change atf-sh to use atf-exec's -t option to control the test case's
       timeouts, instead of doing it internally. Same behavior as before, but
       noticeably faster.

     * atf-exec's -g option and atf-killpg are gone due to the previous
       change.

     * Added the atf-check(1) tool, a program that executes a given command
       and checks its exit code against a known value and allows the
       management of stdout and stderr in multiple ways. This replaces the
       previous atf_check function in the atf-sh library and exposes this
       functionality to both atf-c and atf-c++.

     * Added the ATF_REQUIRE family of macros to the C interface. These help
       in checking for fatal test conditions. The old ATF_CHECK macros now
       perform non-fatal checks only. I.e. by using ATF_CHECK, the test case
       can now continue its execution and the failures will not be reported
       until the end of the whole run.

     * Extended the amount of ATF_CHECK_* C macros with new ones to provide
       more features to the developer. These also have their corresponding
       counterparts in the ATF_REQUIRE_* family. The new macros (listing the
       suffixes only) are: _EQ (replaces _EQUAL), _EQ_MSG, _STREQ and
       _STREQ_MSG.

                             Changes in version 0.5

   Experimental version released on May 1st, 2008.

     * Clauses 3 and 4 of the BSD license used by the project were dropped.
       All the code is now under a 2-clause BSD license compatible with the
       GNU General Public License (GPL).

     * Added a C-only binding so that binary test programs do not need to be
       tied to C++ at all. This binding is now known as the atf-c library.

     * Renamed the C++ binding to atf-c++ for consistency with the new atf-c.

     * Renamed the POSIX shell binding to atf-sh for consistency with the new
       atf-c and atf-c++.

     * Added a -w flag to test programs through which it is possible to
       specify the work directory to be used. This was possible in prior
       releases by defining the workdir configuration variable (-v
       workdir=...), but was a conceptually incorrect mechanism.

     * Test programs now preserve the execution order of test cases when they
       are given in the command line. Even those mentioned more than once are
       executed multiple times to comply with the user's requests.

                             Changes in version 0.4

   Experimental version released on February 4th, 2008.

     * Added two new manual pages, atf-c++-api and atf-sh-api, describing the
       C++ and POSIX shell interfaces used to write test programs.

     * Added a pkg-config file, useful to get the flags to build against the
       C++ library or to easily detect the presence of ATF.

     * Added a way for test cases to require a specific architecture and/or
       machine type through the new 'require.arch' and 'require.machine'
       meta-data properties, respectively.

     * Added the 'timeout' property to test cases, useful to set an
       upper-bound limit for the test's run time and thus prevent global test
       program stalls due to the test case's misbehavior.

     * Added the atf-exec(1) internal utility, used to execute a command
       after changing the process group it belongs to.

     * Added the atf-killpg(1) internal utility, used to kill process groups.

     * Multiple portability fixes. Of special interest, full support for
       SunOS (Solaris Express Developer Edition 2007/09) using the Sun Studio
       12 C++ compiler.

     * Fixed a serious bug that prevented atf-run(1) from working at all
       under Fedora 8 x86_64. Due to the nature of the bug, other platforms
       were likely affected too.

                             Changes in version 0.3

   Experimental version released on November 11th, 2007.

     * Added XML output support to atf-report. This is accompanied by a DTD
       for the format's structure and sample XSLT/CSS files to post-process
       this output and convert it to a plain HTML report.

     * Changed atf-run to add system information to the report it generates.
       This is currently used by atf-report's XML output only, and is later
       printed in the HTML reports in a nice and useful summary table. The
       user and system administrator are allowed to tune this feature by
       means of hooks.

     * Removed the test cases' 'isolated' property. This was intended to
       avoid touching the file system at all when running the related test
       case, but this has not been true for a long while: some control files
       are unconditionally required for several purposes, and we cannot
       easily get rid of them. This way we remove several critical and
       delicate pieces of code.

     * Improved atf-report's CSV output format to include information about
       test programs too.

     * Fixed the tests that used atf-compile to not require this tool as a
       helper. Avoids systems without build-time utilities to skip many tests
       that could otherwise be run. (E.g. NetBSD without the comp.tgz set
       installed.)

     * Many general cleanups: Fixed many pieces of code marked as ugly and/or
       incomplete.

                             Changes in version 0.2

   Experimental version released on September 20th, 2007.

     * Test cases now get a known umask on entry.

     * atf-run now detects many unexpected failures caused by test programs
       and reports them as bogus tests. atf-report is able to handle these
       new errors and nicely reports them to the user.

     * All the data formats read and written by the tools have been
       documented and cleaned up. These include those grammars that define
       how the different components communicate with each other as well as
       the format of files written by the developers and users: the Atffiles
       and the configuration files.

     * Added the atf-version tool, a utility that displays information about
       the currently installed version of ATF.

     * Test cases can now define an optional cleanup routine to undo their
       actions regardless of their exit status.

     * atf-report now summarizes the list of failed (bogus) test programs
       when using the ticker output format.

     * Test programs now capture some termination signals and clean up any
       temporary files before exiting the program.

     * Multiple bug fixes and improvements all around.

                             Changes in version 0.1

   Experimental version released on August 20th, 2007.

     * First public version. This was released coinciding with the end of the
       Google Summer of Code 2007 program.