Home | History | Annotate | Line # | Download | only in testsuite
      1   1.1  christos This is a collection of tests for GDB.
      2   1.1  christos 
      3   1.1  christos The file gdb/README contains basic instructions on how to run the
      4   1.1  christos testsuite, while this file documents additional options and controls
      5   1.1  christos that are available.  The GDB wiki may also have some pages with ideas
      6   1.1  christos and suggestions.
      7   1.1  christos 
      8   1.1  christos 
      9   1.1  christos Running the Testsuite
     10   1.1  christos *********************
     11   1.1  christos 
     12   1.1  christos There are two ways to run the testsuite and pass additional parameters
     13   1.1  christos to DejaGnu.  The first is to do `make check' in the main build
     14   1.1  christos directory and specifying the makefile variable `RUNTESTFLAGS':
     15   1.1  christos 
     16   1.9  christos 	 make check RUNTESTFLAGS='GDB=/usr/bin/gdb gdb.base/a2-run.exp'
     17   1.1  christos 
     18   1.1  christos The second is to cd to the testsuite directory and invoke the DejaGnu
     19   1.1  christos `runtest' command directly.
     20   1.1  christos 
     21   1.1  christos 	cd testsuite
     22   1.1  christos 	make site.exp
     23   1.9  christos 	runtest GDB=/usr/bin/gdb
     24   1.1  christos 
     25   1.1  christos (The `site.exp' file contains a handful of useful variables like host
     26   1.1  christos and target triplets, and pathnames.)
     27   1.1  christos 
     28   1.6  christos Parallel testing
     29   1.6  christos ****************
     30   1.6  christos 
     31   1.6  christos If not testing with a remote host (in DejaGnu's sense), you can run
     32   1.6  christos the GDB test suite in a fully parallel mode.  In this mode, each .exp
     33   1.6  christos file runs separately and maybe simultaneously.  The test suite ensures
     34   1.6  christos that all the temporary files created by the test suite do not clash,
     35   1.6  christos by putting them into separate directories.  This mode is primarily
     36   1.6  christos intended for use by the Makefile.
     37   1.6  christos 
     38   1.6  christos For GNU make, the Makefile tries to run the tests in parallel mode if
     39   1.6  christos any -j option is given.  For a non-GNU make, tests are not
     40   1.6  christos parallelized.
     41   1.6  christos 
     42   1.6  christos If RUNTESTFLAGS is not empty, then by default the tests are
     43   1.6  christos serialized.  This can be overridden by either using the
     44   1.6  christos `check-parallel' target in the Makefile, or by setting FORCE_PARALLEL
     45   1.6  christos to any non-empty value:
     46   1.6  christos 
     47   1.6  christos 	make check-parallel RUNTESTFLAGS="--target_board=native-gdbserver"
     48   1.6  christos 	make check RUNTESTFLAGS="--target_board=native-gdbserver" FORCE_PARALLEL=1
     49   1.6  christos 
     50   1.6  christos If you want to use runtest directly instead of using the Makefile, see
     51   1.6  christos the description of GDB_PARALLEL below.
     52   1.6  christos 
     53   1.6  christos Racy testcases
     54   1.6  christos **************
     55   1.6  christos 
     56   1.6  christos Sometimes, new testcases are added to the testsuite that are not
     57   1.6  christos entirely deterministic, and can randomly pass or fail.  We call them
     58   1.6  christos "racy testcases", and they can be bothersome when one is comparing
     59   1.6  christos different testsuite runs.  In order to help identifying them, it is
     60   1.6  christos possible to run the tests several times in a row and ask the testsuite
     61   1.6  christos machinery to analyze the results.  To do that, you need to specify the
     62   1.6  christos RACY_ITER environment variable to make:
     63   1.6  christos 
     64   1.6  christos 	make check RACY_ITER=5 -j4
     65   1.6  christos 
     66   1.6  christos The value assigned to RACY_ITER represents the number of times you
     67   1.6  christos wish to run the tests in sequence (in the example above, the entire
     68   1.6  christos testsuite will be executed 5 times in a row, in parallel).  It is also
     69   1.6  christos possible to check just a specific test:
     70   1.6  christos 
     71   1.6  christos 	make check TESTS='gdb.base/default.exp' RACY_ITER=3
     72   1.6  christos 
     73   1.6  christos One can also decide to call the Makefile rules by hand inside the
     74   1.6  christos gdb/testsuite directory, e.g.:
     75   1.6  christos 
     76  1.10  christos 	make check-parallel-racy -j4
     77   1.6  christos 
     78   1.6  christos In which case the value of the DEFAULT_RACY_ITER variable (inside
     79   1.6  christos gdb/testsuite/Makefile.in) will be used to determine how many
     80   1.6  christos iterations will be run.
     81   1.6  christos 
     82   1.6  christos After running the tests, you shall see a file name 'racy.sum' in the
     83   1.6  christos gdb/testsuite directory.  You can also inspect the generated *.log and
     84   1.6  christos *.sum files by looking into the gdb/testsuite/racy_ouputs directory.
     85   1.6  christos 
     86   1.6  christos If you already have *.sum files generated from previous testsuite runs
     87   1.6  christos and you would like to analyze them without having to run the testsuite
     88   1.6  christos again, you can also use the 'analyze-racy-logs.py' script directly.
     89   1.6  christos It is located in the gdb/testsuite/ directory, and it expects a list
     90   1.6  christos of two or more *.sum files to be provided as its argument.  For
     91   1.6  christos example:
     92   1.6  christos 
     93   1.6  christos 	./gdb/testsuite/analyze-racy-logs.py testsuite-01/gdb.sum \
     94   1.6  christos 	  testsuite-02/gdb.sum testsuite-03/gdb.sum
     95   1.6  christos 
     96   1.6  christos The script will output its analysis report to the standard output.
     97   1.6  christos 
     98   1.9  christos Re-running Tests Outside The Testsuite
     99   1.9  christos **************************************
    100   1.9  christos 
    101   1.9  christos When running a test, the arguments used to run GDB are saved to gdb.cmd and
    102   1.9  christos all commands sent to GDB are saved to gdb.in.  As well as being a reference
    103   1.9  christos of the commands run, they can be used to manually re-run a test by using
    104   1.9  christos the gdb.in file as a batch file to a GDB launched with the arguments in the
    105   1.9  christos gdb.cmd file, for example:
    106   1.9  christos 	$(cat outputs/gdb.base/store/gdb.cmd) -x outputs/gdb.base/store/gdb.in
    107   1.9  christos 
    108   1.9  christos Tests that run GDB multiple times will append .1, .2, .3 etc to the end
    109   1.9  christos of each .cmd and .in file.
    110   1.9  christos 
    111   1.9  christos When gdbserver is launched as part of a test, a gdbserver.cmd will be created.
    112   1.9  christos To re-run these tests, run the contents of gdbserver.cmd in a separate
    113   1.9  christos terminal before running gdb, for example:
    114   1.9  christos 	$(cat outputs/gdb.base/store/gdbserver.cmd)
    115   1.9  christos Alternatively, if the test is run with GDBSERVER_DEBUG="replay", then this
    116   1.9  christos will create a gdbserver.replay file which can be used with the gdbreplay tool,
    117   1.9  christos instead of launching gdbserver.
    118   1.9  christos 
    119   1.1  christos Running the Performance Tests
    120   1.1  christos *****************************
    121   1.1  christos 
    122   1.1  christos GDB Testsuite includes performance test cases, which are not run together
    123   1.1  christos with other test cases, because performance test cases are slow and need
    124   1.1  christos a quiet system.  There are two ways to run the performance test cases.
    125   1.1  christos The first is to do `make check-perf' in the main build directory:
    126   1.1  christos 
    127   1.1  christos 	make check-perf RUNTESTFLAGS="solib.exp SOLIB_COUNT=8"
    128   1.1  christos 
    129   1.1  christos The second is to cd to the testsuite directory and invoke the DejaGnu
    130   1.1  christos `runtest' command directly.
    131   1.1  christos 
    132   1.1  christos 	cd testsuite
    133   1.1  christos 	make site.exp
    134   1.1  christos 	runtest GDB_PERFTEST_MODE=both GDB_PERFTEST_TIMEOUT=4000 --directory=gdb.perf solib.exp SOLIB_COUNT=8
    135   1.1  christos 
    136   1.1  christos Only "compile", "run" and "both" are valid to GDB_PERFTEST_MODE.  They
    137   1.1  christos stand for "compile tests only", "run tests only", and "compile and run
    138   1.1  christos tests" respectively.  "both" is the default.  GDB_PERFTEST_TIMEOUT
    139   1.1  christos specify the timeout, which is 3000 in default.  The result of
    140   1.1  christos performance test is appended in `testsuite/perftest.log'.
    141   1.1  christos 
    142   1.1  christos Testsuite Parameters
    143   1.1  christos ********************
    144   1.1  christos 
    145   1.1  christos The following parameters are DejaGNU variables that you can set to
    146   1.1  christos affect the testsuite run globally.
    147   1.1  christos 
    148   1.1  christos GDB
    149   1.1  christos 
    150   1.1  christos By default, the testsuite exercises the GDB in the build directory,
    151   1.1  christos but you can set GDB to be a pathname to a different version.  For
    152   1.1  christos instance,
    153   1.1  christos 
    154   1.1  christos     make check RUNTESTFLAGS=GDB=/usr/bin/gdb
    155   1.1  christos 
    156   1.1  christos runs the testsuite on the GDB in /usr/bin.
    157   1.1  christos 
    158   1.1  christos GDBSERVER
    159   1.1  christos 
    160   1.1  christos You can set GDBSERVER to be a particular GDBserver of interest, so for
    161   1.1  christos instance
    162   1.1  christos 
    163   1.1  christos     make check RUNTESTFLAGS="GDB=/usr/bin/gdb GDBSERVER=/usr/bin/gdbserver"
    164   1.1  christos 
    165   1.1  christos checks both the installed GDB and GDBserver.
    166   1.1  christos 
    167  1.10  christos GDB_DATA_DIRECTORY
    168  1.10  christos 
    169  1.10  christos If you set GDB, then by default the testsuite assumes you are
    170  1.10  christos exercising an installed GDB, and thus the testsuite lets GDB use its
    171  1.10  christos configured data directory.  Otherwise, if you don't set GDB, then by
    172  1.10  christos default the tested GDB uses the data directory found under the GDB
    173  1.10  christos build directory.  You can override this by setting GDB_DATA_DIRECTORY.
    174  1.10  christos For instance:
    175  1.10  christos 
    176  1.10  christos     make check \
    177  1.10  christos       RUNTESTFLAGS="GDB=/path/to/other/build/gdb \
    178  1.10  christos                     GDB_DATA_DIRECTORY=/path/to/other/build/gdb/data-directory"
    179  1.10  christos 
    180   1.1  christos INTERNAL_GDBFLAGS
    181   1.1  christos 
    182   1.1  christos Command line options passed to all GDB invocations.
    183  1.10  christos The default is set in lib/gdb.exp.
    184   1.1  christos 
    185   1.1  christos This is actually considered an internal variable, and you
    186   1.1  christos won't normally want to change it.  However, in some situations,
    187   1.1  christos this may be tweaked as a last resort if the testsuite doesn't
    188   1.1  christos have direct support for the specifics of your environment.
    189   1.1  christos The testsuite does not override a value provided by the user.
    190   1.1  christos 
    191   1.1  christos As an example, when testing an installed GDB that has been
    192   1.1  christos configured with `--with-system-gdbinit', like by default,
    193   1.1  christos you do not want ~/.gdbinit to interfere with tests, but, you
    194   1.1  christos may want the system .gdbinit file loaded.  As there's no way to
    195   1.1  christos ask the testsuite, or GDB, to load the system gdbinit but
    196   1.1  christos not ~/.gdbinit, a workaround is then to remove `-nx' from
    197   1.1  christos INTERNAL_GDBFLAGS, and point $HOME at a directory without
    198   1.1  christos a .gdbinit.  For example:
    199   1.1  christos 
    200   1.1  christos 	cd testsuite
    201   1.1  christos 	HOME=`pwd` runtest \
    202   1.1  christos 	  GDB=/usr/bin/gdb \
    203   1.1  christos 	  GDBSERVER=/usr/bin/gdbserver \
    204  1.11  christos 	  INTERNAL_GDBFLAGS="-nw -q -iex 'set height 0' -iex 'set width 0'"
    205  1.10  christos 
    206  1.10  christos Note that we do not need to specify '-data-directory' here
    207  1.10  christos as we are testing an installed GDB.
    208   1.1  christos 
    209   1.1  christos GDB_PARALLEL
    210   1.1  christos 
    211   1.9  christos To use parallel testing mode without using the Makefile, set
    212   1.6  christos GDB_PARALLEL on the runtest command line to "yes".  Before starting
    213   1.6  christos the tests, you must ensure that the directories cache, outputs, and
    214   1.6  christos temp in the test suite build directory are either empty or have been
    215   1.6  christos deleted.  cache in particular is used to share data across invocations
    216   1.6  christos of runtest, and files there may affect the test results.  The Makefile
    217   1.6  christos automatically does these deletions.
    218   1.6  christos 
    219   1.6  christos FORCE_PARALLEL
    220   1.6  christos 
    221   1.6  christos Setting FORCE_PARALLEL to any non-empty value forces parallel testing
    222   1.6  christos mode even if RUNTESTFLAGS is not empty.
    223   1.6  christos 
    224   1.6  christos FORCE_SEPARATE_MI_TTY
    225   1.6  christos 
    226   1.6  christos Setting FORCE_MI_SEPARATE_UI to 1 forces all MI testing to start GDB
    227   1.6  christos in console mode, with MI running on a separate TTY, on a secondary UI
    228   1.6  christos started with "new-ui".
    229   1.1  christos 
    230   1.1  christos GDB_INOTIFY
    231   1.1  christos 
    232   1.1  christos For debugging parallel mode, it is handy to be able to see when a test
    233   1.1  christos case writes to a file outside of its designated output directory.
    234   1.1  christos 
    235   1.1  christos If you have the inotify-tools package installed, you can set the
    236   1.1  christos GDB_INOTIFY variable on the runtest command line.  This will cause the
    237   1.1  christos test suite to watch for parallel-unsafe file creations and report
    238   1.1  christos them, both to stdout and in the test suite log file.
    239   1.1  christos 
    240   1.1  christos This setting is only meaningful in conjunction with GDB_PARALLEL.
    241   1.1  christos 
    242   1.3  christos TESTS
    243   1.3  christos 
    244   1.3  christos This variable is used to specify which set of tests to run.
    245   1.3  christos It is passed to make (not runtest) and its contents are a space separated
    246   1.3  christos list of tests to run.
    247   1.3  christos 
    248   1.3  christos If using GNU make then the contents are wildcard-expanded using
    249   1.3  christos GNU make's $(wildcard) function.  Test paths must be fully specified,
    250   1.3  christos relative to the "testsuite" subdirectory.  This allows one to run all
    251   1.3  christos tests in a subdirectory by passing "gdb.subdir/*.exp", or more simply
    252   1.3  christos by using the check-gdb.subdir target in the Makefile.
    253   1.3  christos 
    254   1.3  christos If for some strange reason one wanted to run all tests that begin with
    255   1.3  christos the letter "d" that is also possible: TESTS="*/d*.exp".
    256   1.3  christos 
    257   1.3  christos Do not write */*.exp to specify all tests (assuming all tests are only
    258   1.3  christos nested one level deep, which is not necessarily true).  This will pick up
    259   1.3  christos .exp files in ancillary directories like "lib" and "config".
    260   1.3  christos Instead write gdb.*/*.exp.
    261   1.3  christos 
    262   1.3  christos Example:
    263   1.3  christos 
    264   1.3  christos 	make -j10 check TESTS="gdb.server/[s-w]*.exp */x*.exp"
    265   1.3  christos 
    266   1.3  christos If not using GNU make then the value is passed directly to runtest.
    267   1.3  christos If not specified, all tests are run.
    268   1.3  christos 
    269   1.3  christos READ1
    270   1.3  christos 
    271   1.3  christos This make (not runtest) variable is used to specify whether the
    272   1.3  christos testsuite preloads the read1.so library into expect.  Any non-empty
    273   1.3  christos value means true.  See "Race detection" below.
    274   1.3  christos 
    275   1.8  christos GDB_TEST_SOCKETHOST
    276   1.8  christos 
    277   1.8  christos This variable can provide the hostname/address that should be used
    278   1.8  christos when performing GDBserver-related tests.  This is useful in some
    279   1.8  christos situations, e.g., when you want to test the IPv6 connectivity of GDB
    280   1.8  christos and GDBserver, or when using a different hostname/address is needed.
    281   1.8  christos For example, to make GDB and GDBserver use IPv6-only connections, you
    282   1.8  christos can do:
    283   1.8  christos 
    284   1.8  christos 	make check TESTS="gdb.server/*.exp" RUNTESTFLAGS='GDB_TEST_SOCKETHOST=tcp6:[::1]'
    285   1.8  christos 
    286   1.8  christos Note that only a hostname/address can be provided, without a port
    287   1.8  christos number.
    288   1.8  christos 
    289   1.8  christos TS
    290   1.8  christos 
    291   1.8  christos This variable turns on the timestamp printing for each line of "make
    292   1.8  christos check".  Note that the timestamp will be printed on stdout output
    293   1.8  christos only.  In other words, there will be no timestamp output on either
    294   1.8  christos gdb.sum and gdb.log files.  If you would like to enable timestamp
    295   1.8  christos printing, you can do:
    296   1.8  christos 
    297   1.8  christos 	make check TS=1
    298   1.8  christos 
    299   1.8  christos TS_FORMAT
    300   1.8  christos 
    301   1.8  christos You can provide a custom format for timestamp printing with this
    302   1.8  christos variable.  The format must be a string compatible with "strftime".
    303   1.8  christos This variable is only useful when the TS variable is also provided.
    304   1.8  christos If you would like to change the output format of the timestamp, you
    305   1.8  christos can do:
    306   1.8  christos 
    307   1.8  christos 	make check TS=1 TS_FORMAT='[%b %H:%S]'
    308   1.8  christos 
    309   1.9  christos GDB_DEBUG
    310   1.9  christos 
    311   1.9  christos When set gdb debug is sent to the file gdb.debug in the test output
    312   1.9  christos directory.  It should be set to a comma separated list of gdb debug
    313   1.9  christos components.
    314   1.9  christos For example, to turn on debugging for infrun and target, you can do:
    315   1.9  christos 
    316   1.9  christos 	make check GDB_DEBUG="infrun,target"
    317   1.9  christos 
    318   1.9  christos GDBSERVER_DEBUG
    319   1.9  christos 
    320   1.9  christos When set gdbserver debug is sent to the a file in the test output directory.
    321   1.9  christos It should be set to a comma separated list of the following options:
    322   1.9  christos 	debug  - write gdbserver debug to gdbserver.debug.
    323   1.9  christos 	remote - write gdbserver remote debug to gdbserver.debug.
    324   1.9  christos 	replay - write a replay log to the file gdbserver.replay for use
    325   1.9  christos 		 with gdbreplay.
    326   1.9  christos Alternatively, it can be set to "all" to turn on all the above
    327   1.9  christos For example, to turn on gdbserver debugging, you can do:
    328   1.9  christos 
    329   1.9  christos 	make check GDBSERVER_DEBUG="debug,replay"
    330   1.9  christos 
    331  1.11  christos GDB_TARGET_USERNAME
    332  1.11  christos GDB_HOST_USERNAME
    333  1.11  christos 
    334  1.11  christos These settings are only used with the check-all-boards target, and
    335  1.11  christos should be the usernames of two separate users on the local machine,
    336  1.11  christos both of which the current user can ssh to without a password.
    337  1.11  christos 
    338  1.11  christos These users will be used by board files that simulate remote targets
    339  1.11  christos by switching to a different user on the same machine.  These users
    340  1.11  christos will have random files copied into their $HOME directories, so it is a
    341  1.11  christos good idea to setup new users just for this purpose.
    342  1.11  christos 
    343  1.11  christos Testing All Simple Boards
    344  1.11  christos *************************
    345  1.11  christos 
    346  1.11  christos There are a number of boards that ship with GDB that simulate common
    347  1.11  christos debug scenarios.  For example by sshing to a different user on the
    348  1.11  christos local machine and running gdbserver as this alternative user we aim to
    349  1.11  christos simulate a true remote debug experience.
    350  1.11  christos 
    351  1.11  christos There is a script binutils-gdb/gdb/testssuite/make-check-all.sh which
    352  1.11  christos can be used to run a defined set of tests using all of the available
    353  1.11  christos simple board files.  Support for using this script is also included in
    354  1.11  christos GDB's makefile, and can be used as:
    355  1.11  christos 
    356  1.11  christos   make check-all-boards GDB_TARGET_USERNAME=remote-target \
    357  1.11  christos 			GDB_HOST_USERNAME=remote-host \
    358  1.11  christos 			TESTS="gdb.base/break.exp"
    359  1.11  christos 
    360  1.11  christos The 'remote-target' and 'remote-host' can be replaced with any user
    361  1.11  christos names on the local machine, the only requirements are that the current
    362  1.11  christos user must be able to ssh to these users without a password, and these
    363  1.11  christos users must be happy to have arbitrary files copied into their $HOME
    364  1.11  christos directory.  Ideally, these users would be setup just for GDB testing.
    365  1.11  christos 
    366  1.11  christos The check-all-boards target requires that TESTS be defined, though it
    367  1.11  christos is fine to include multiple tests.
    368  1.11  christos 
    369  1.11  christos The results are preserved, and can be found in the directory
    370  1.11  christos gdb/testsuite/check-all/.  The results are split by the board file
    371  1.11  christos used.
    372  1.11  christos 
    373  1.10  christos Architecture-specific Parameters
    374  1.10  christos ******************************
    375  1.10  christos 
    376  1.10  christos This section documents architecture-specific parameters that can be used with
    377  1.10  christos the GDB testsuite.
    378  1.10  christos 
    379  1.10  christos - AArch64 (Linux)
    380  1.10  christos 
    381  1.10  christos ARM_CC_FOR_TARGET
    382  1.10  christos 
    383  1.10  christos The AArch64 ports of GDB and GDBserver support debugging AArch32
    384  1.10  christos 32-bit programs running on 64-bit state.  There are some tests under
    385  1.10  christos gdb.multi/ that exercise this particular feature.
    386  1.10  christos 
    387  1.10  christos By default, the testsuite tries to find a compiler capable of
    388  1.10  christos generating 32-bit executables.  If no compiler is found, or if the
    389  1.10  christos 32-bit executable generated by the found compiler can't be executed
    390  1.10  christos correctly, the tests will be marked UNSUPPORTED.  The list of 32-bit
    391  1.10  christos Arm compiler names the testsuite will try can be found in
    392  1.10  christos gdb/testsuite/lib/gdb.exp:arm_cc_for_target.
    393  1.10  christos 
    394  1.10  christos You can set ARM_CC_FOR_TARGET to override the search and explicitly
    395  1.10  christos specify the compiler to use.  This variable should contain the command
    396  1.10  christos line for the compiler, including the full path to it, if the compiler
    397  1.10  christos is not in $PATH.
    398  1.10  christos 
    399  1.10  christos Example:
    400  1.10  christos 
    401  1.10  christos 	make check-gdb TESTS="gdb.multi/multi-arch.exp" RUNTESTFLAGS="ARM_CC_FOR_TARGET=arm-linux-gnueabihf-gcc"
    402  1.10  christos 
    403   1.3  christos Race detection
    404   1.3  christos **************
    405   1.3  christos 
    406   1.3  christos The testsuite includes a mechanism that helps detect test races.
    407   1.3  christos 
    408   1.3  christos For example, say the program running under expect outputs "abcd", and
    409   1.3  christos a test does something like this:
    410   1.3  christos 
    411   1.3  christos   expect {
    412   1.3  christos     "a.*c" {
    413   1.3  christos     }
    414   1.3  christos     "b" {
    415   1.3  christos     }
    416   1.3  christos     "a" {
    417   1.3  christos     }
    418   1.3  christos   }
    419   1.3  christos 
    420   1.3  christos Which case happens to match depends on what expect manages to read
    421   1.3  christos into its internal buffer in one go.  If it manages to read three bytes
    422   1.3  christos or more, then the first case matches.  If it manages to read two
    423   1.3  christos bytes, then the second case matches.  If it manages to read only one
    424   1.3  christos byte, then the third case matches.
    425   1.3  christos 
    426   1.3  christos To help detect these cases, the race detection mechanism preloads a
    427   1.3  christos library into expect that forces the `read' system call to always
    428   1.3  christos return at most 1 byte.
    429   1.3  christos 
    430   1.3  christos To enable this, either pass a non-empty value in the READ1 make
    431   1.3  christos variable, or use the check-read1 make target instead of check.
    432   1.3  christos 
    433  1.10  christos Example:
    434   1.3  christos 
    435   1.3  christos 	make -j10 check-read1 TESTS="*/paginate-*.exp"
    436  1.10  christos 
    437  1.10  christos If you've already built the read1 support code, either via a previous
    438  1.10  christos 'check-read1' run, or by using "make read1", you can use:
    439  1.10  christos 
    440   1.3  christos 	make -j10 check READ1="1"
    441   1.1  christos 
    442   1.9  christos Note: While the intention is to detect races and make otherwise passing tests
    443   1.9  christos fail, it can also have the effect of making otherwise failing tests pass.
    444   1.9  christos This happens f.i. if the test is trying to match a gdb prompt using an end of
    445   1.9  christos input marker "${gdb_prompt} $" and there is output after the gdb prompt.  This
    446   1.9  christos may either pass or fail in normal operation, but using check-read1 will ensure
    447  1.10  christos that it passes.  Use check-readmore to detect this type of failure.
    448   1.9  christos 
    449   1.1  christos Testsuite Configuration
    450   1.1  christos ***********************
    451   1.1  christos 
    452   1.1  christos It is possible to adjust the behavior of the testsuite by defining
    453   1.1  christos the global variables listed below, either in a `site.exp' file,
    454   1.1  christos or in a board file.
    455   1.1  christos 
    456   1.1  christos gdb_test_timeout
    457   1.1  christos 
    458   1.1  christos Defining this variable changes the default timeout duration used
    459   1.1  christos during communication with GDB.  More specifically, the global variable
    460   1.1  christos used during testing is `timeout', but this variable gets reset to
    461   1.1  christos `gdb_test_timeout' at the beginning of each testcase, which ensures
    462   1.1  christos that any local change to `timeout' in a testcase does not affect
    463   1.1  christos subsequent testcases.
    464   1.1  christos 
    465   1.1  christos This global variable comes in handy when the debugger is slower than
    466   1.1  christos normal due to the testing environment, triggering unexpected `TIMEOUT'
    467   1.1  christos test failures.  Examples include when testing on a remote machine, or
    468   1.1  christos against a system where communications are slow.
    469   1.1  christos 
    470   1.1  christos If not specifically defined, this variable gets automatically defined
    471   1.1  christos to the same value as `timeout' during the testsuite initialization.
    472   1.1  christos The default value of the timeout is defined in the file
    473   1.1  christos `testsuite/config/unix.exp' (at least for Unix hosts; board files may
    474   1.1  christos have their own values).
    475   1.1  christos 
    476   1.3  christos gdb_reverse_timeout
    477   1.3  christos 
    478   1.3  christos Defining this variable changes the default timeout duration when tests
    479   1.3  christos under gdb.reverse directory are running.  Process record and reverse
    480   1.3  christos debugging is so slow that its tests have unexpected `TIMEOUT' test
    481   1.3  christos failures.  This global variable is useful to bump up the value of
    482   1.3  christos `timeout' for gdb.reverse tests and doesn't cause any delay where
    483   1.3  christos actual failures happen in the rest of the testsuite.
    484   1.3  christos 
    485   1.1  christos 
    486   1.1  christos Board Settings
    487   1.1  christos **************
    488   1.1  christos 
    489   1.1  christos DejaGNU includes the concept of a "board file", which specifies
    490   1.1  christos testing details for a particular target (which are often bare circuit
    491   1.1  christos boards, thus the name).
    492   1.1  christos 
    493   1.1  christos In the GDB testsuite specifically, the board file may include a
    494   1.1  christos number of "board settings" that test cases may check before deciding
    495   1.1  christos whether to exercise a particular feature.  For instance, a board
    496   1.1  christos lacking any I/O devices, or perhaps simply having its I/O devices
    497   1.1  christos not wired up, should set `noinferiorio'.
    498   1.1  christos 
    499   1.1  christos Here are the supported board settings:
    500   1.1  christos 
    501   1.1  christos gdb,cannot_call_functions
    502   1.1  christos 
    503   1.1  christos   The board does not support inferior call, that is, invoking inferior
    504   1.1  christos   functions in GDB.
    505   1.1  christos 
    506   1.1  christos gdb,can_reverse
    507   1.1  christos 
    508   1.1  christos   The board supports reverse execution.
    509   1.1  christos 
    510   1.1  christos gdb,no_hardware_watchpoints
    511   1.1  christos 
    512   1.1  christos   The board does not support hardware watchpoints.
    513   1.1  christos 
    514   1.1  christos gdb,nofileio
    515   1.1  christos 
    516   1.1  christos   GDB is unable to intercept target file operations in remote and
    517   1.1  christos   perform them on the host.
    518   1.1  christos 
    519   1.1  christos gdb,noinferiorio
    520   1.1  christos 
    521   1.1  christos   The board is unable to provide I/O capability to the inferior.
    522   1.1  christos 
    523   1.1  christos gdb,noresults
    524   1.1  christos 
    525   1.1  christos   A program will not return an exit code or result code (or the value
    526   1.1  christos   of the result is undefined, and should not be looked at).
    527   1.1  christos 
    528   1.1  christos gdb,nosignals
    529   1.1  christos 
    530   1.1  christos   The board does not support signals.
    531   1.1  christos 
    532   1.1  christos gdb,skip_huge_test
    533   1.1  christos 
    534   1.1  christos   Skip time-consuming tests on the board with slow connection.
    535   1.1  christos 
    536   1.1  christos gdb,skip_float_tests
    537   1.1  christos 
    538   1.1  christos   Skip tests related to floating point.
    539   1.1  christos 
    540   1.1  christos gdb,use_precord
    541   1.1  christos 
    542   1.1  christos   The board supports process record.
    543   1.1  christos 
    544   1.3  christos gdb_init_command
    545   1.3  christos gdb_init_commands
    546   1.3  christos 
    547   1.3  christos   Commands to send to GDB every time a program is about to be run.  The
    548   1.3  christos   first of these settings defines a single command as a string.  The
    549   1.3  christos   second defines a TCL list of commands being a string each.  The commands
    550   1.3  christos   are sent one by one in a sequence, first from `gdb_init_command', if any,
    551   1.3  christos   followed by individual commands from `gdb_init_command', if any, in this
    552   1.3  christos   list's order.
    553   1.3  christos 
    554   1.1  christos gdb_server_prog
    555   1.1  christos 
    556   1.1  christos   The location of GDBserver.  If GDBserver somewhere other than its
    557   1.1  christos   default location is used in test, specify the location of GDBserver in
    558   1.1  christos   this variable.  The location is a file name for GDBserver, and may be
    559   1.1  christos   either absolute or relative to the testsuite subdirectory of the build
    560   1.1  christos   directory.
    561   1.1  christos 
    562   1.1  christos in_proc_agent
    563   1.1  christos 
    564   1.1  christos   The location of the in-process agent (used for fast tracepoints and
    565   1.1  christos   other special tests).  If the in-process agent of interest is anywhere
    566   1.1  christos   other than its default location, set this variable.  The location is a
    567   1.1  christos   filename, and may be either absolute or relative to the testsuite
    568   1.1  christos   subdirectory of the build directory.
    569   1.1  christos 
    570   1.1  christos noargs
    571   1.1  christos 
    572   1.1  christos   GDB does not support argument passing for inferior.
    573   1.1  christos 
    574   1.1  christos no_long_long
    575   1.1  christos 
    576   1.1  christos   The board does not support type long long.
    577   1.1  christos 
    578   1.1  christos use_cygmon
    579   1.1  christos 
    580   1.1  christos   The board is running the monitor Cygmon.
    581   1.1  christos 
    582   1.1  christos use_gdb_stub
    583   1.1  christos 
    584   1.1  christos   The tests are running with a GDB stub.
    585   1.1  christos 
    586   1.1  christos exit_is_reliable
    587   1.1  christos 
    588   1.1  christos   Set to true if GDB can assume that letting the program run to end
    589   1.1  christos   reliably results in program exits being reported as such, as opposed
    590   1.1  christos   to, e.g., the program ending in an infinite loop or the board
    591   1.1  christos   crashing/resetting.  If not set, this defaults to $use_gdb_stub.  In
    592   1.1  christos   other words, native targets are assumed reliable by default, and
    593   1.1  christos   remote stubs assumed unreliable.
    594   1.1  christos 
    595   1.1  christos gdb,predefined_tsv
    596   1.1  christos 
    597   1.1  christos   The predefined trace state variables the board has.
    598   1.1  christos 
    599   1.6  christos gdb,no_thread_names
    600   1.6  christos 
    601   1.6  christos   The target doesn't support thread names.
    602   1.1  christos 
    603   1.9  christos gdb,pie_flag
    604   1.9  christos 
    605   1.9  christos   The flag required to force the compiler to produce position-independent
    606   1.9  christos   executables.
    607   1.9  christos 
    608   1.9  christos gdb,pie_ldflag
    609   1.9  christos 
    610   1.9  christos   The flag required to force the linker to produce position-independent
    611   1.9  christos   executables.
    612   1.9  christos 
    613   1.8  christos gdb,nopie_flag
    614   1.8  christos 
    615   1.8  christos   The flag required to force the compiler to produce non-position-independent
    616   1.8  christos   executables.
    617   1.8  christos 
    618  1.10  christos gdb,nopie_ldflag
    619  1.10  christos 
    620  1.10  christos   The flag required to force the linker to produce non-position-independent
    621  1.10  christos   executables.
    622  1.10  christos 
    623   1.9  christos gdb,debug
    624   1.9  christos 
    625   1.9  christos   When set gdb debug is sent to the file gdb.debug in the test output
    626   1.9  christos   directory.  It should be set to a comma separated list of gdb debug
    627   1.9  christos   components. For example, to turn on debugging for infrun and target, set to
    628   1.9  christos   "infrun,target".
    629   1.9  christos 
    630   1.9  christos gdbserver,debug
    631   1.9  christos 
    632   1.9  christos   When set gdbserver debug is sent to the file gdbserver.debug in the test
    633   1.9  christos   output directory.  For valid values see the entry for GDBSERVER_DEBUG.
    634   1.9  christos 
    635   1.1  christos Testsuite Organization
    636   1.1  christos **********************
    637   1.1  christos 
    638   1.1  christos The testsuite is entirely contained in `gdb/testsuite'.  The main
    639   1.1  christos directory of the testsuite includes some makefiles and configury, but
    640   1.1  christos these are minimal, and used for little besides cleaning up, since the
    641   1.1  christos tests themselves handle the compilation of the programs that GDB will
    642   1.1  christos run.
    643   1.1  christos 
    644   1.1  christos The file `testsuite/lib/gdb.exp' contains common utility procs useful
    645   1.1  christos for all GDB tests, while the directory testsuite/config contains
    646   1.1  christos configuration-specific files, typically used for special-purpose
    647   1.1  christos definitions of procs like `gdb_load' and `gdb_start'.
    648   1.1  christos 
    649   1.1  christos The tests themselves are to be found in directories named
    650   1.1  christos 'testsuite/gdb.* and subdirectories of those.  The names of the test
    651   1.1  christos files must always end with ".exp".  DejaGNU collects the test files by
    652   1.1  christos wildcarding in the test directories, so both subdirectories and
    653   1.1  christos individual files typically get chosen and run in alphabetical order.
    654   1.1  christos 
    655   1.1  christos The following lists some notable types of subdirectories and what they
    656   1.1  christos are for.  Since DejaGNU finds test files no matter where they are
    657   1.1  christos located, and since each test file sets up its own compilation and
    658   1.1  christos execution environment, this organization is simply for convenience and
    659   1.1  christos intelligibility.
    660   1.1  christos 
    661   1.1  christos gdb.base
    662   1.1  christos 
    663   1.1  christos This is the base testsuite.  The tests in it should apply to all
    664   1.1  christos configurations of GDB (but generic native-only tests may live here).
    665   1.1  christos The test programs should be in the subset of C that is both valid
    666   1.1  christos ANSI/ISO C, and C++.
    667   1.1  christos 
    668   1.1  christos gdb.<lang>
    669   1.1  christos 
    670   1.1  christos Language-specific tests for any language besides C.  Examples are
    671   1.7  christos gdb.cp for C++ and gdb.rust for Rust.
    672   1.1  christos 
    673   1.1  christos gdb.<platform>
    674   1.1  christos 
    675   1.1  christos Non-portable tests.  The tests are specific to a specific
    676   1.5  christos configuration (host or target), such as eCos.
    677   1.1  christos 
    678   1.1  christos gdb.arch
    679   1.1  christos 
    680   1.1  christos Architecture-specific tests that are (usually) cross-platform.
    681   1.1  christos 
    682   1.1  christos gdb.<subsystem>
    683   1.1  christos 
    684   1.1  christos Tests that exercise a specific GDB subsystem in more depth.  For
    685   1.1  christos instance, gdb.disasm exercises various disassemblers, while
    686   1.1  christos gdb.stabs tests pathways through the stabs symbol reader.
    687   1.1  christos 
    688   1.1  christos gdb.perf
    689   1.1  christos 
    690   1.1  christos GDB performance tests.
    691   1.1  christos 
    692   1.1  christos Writing Tests
    693   1.1  christos *************
    694   1.1  christos 
    695   1.1  christos In many areas, the GDB tests are already quite comprehensive; you
    696   1.1  christos should be able to copy existing tests to handle new cases.  Be aware
    697   1.1  christos that older tests may use obsolete practices but have not yet been
    698   1.1  christos updated.
    699   1.1  christos 
    700   1.1  christos You should try to use `gdb_test' whenever possible, since it includes
    701   1.1  christos cases to handle all the unexpected errors that might happen.  However,
    702   1.1  christos it doesn't cost anything to add new test procedures; for instance,
    703   1.1  christos gdb.base/exprs.exp defines a `test_expr' that calls `gdb_test'
    704   1.1  christos multiple times.
    705   1.1  christos 
    706   1.1  christos Only use `send_gdb' and `gdb_expect' when absolutely necessary.  Even
    707   1.1  christos if GDB has several valid responses to a command, you can use
    708   1.1  christos `gdb_test_multiple'.  Like `gdb_test', `gdb_test_multiple' recognizes
    709   1.1  christos internal errors and unexpected prompts.
    710   1.1  christos 
    711   1.1  christos Do not write tests which expect a literal tab character from GDB.  On
    712   1.1  christos some operating systems (e.g. OpenBSD) the TTY layer expands tabs to
    713   1.1  christos spaces, so by the time GDB's output reaches `expect' the tab is gone.
    714   1.1  christos 
    715   1.1  christos The source language programs do *not* need to be in a consistent
    716   1.1  christos style.  Since GDB is used to debug programs written in many different
    717   1.1  christos styles, it's worth having a mix of styles in the testsuite; for
    718   1.1  christos instance, some GDB bugs involving the display of source lines might
    719   1.1  christos never manifest themselves if the test programs used GNU coding style
    720   1.1  christos uniformly.
    721   1.1  christos 
    722   1.1  christos Some testcase results need more detailed explanation:
    723   1.1  christos 
    724   1.1  christos KFAIL
    725   1.1  christos 
    726   1.1  christos Use KFAIL for known problem of GDB itself.  You must specify the GDB
    727   1.1  christos bug report number, as in these sample tests:
    728   1.1  christos 
    729   1.1  christos 	kfail "gdb/13392" "continue to marker 2"
    730   1.1  christos 
    731   1.1  christos or
    732   1.1  christos 
    733   1.1  christos 	setup_kfail gdb/13392 "*-*-*"
    734   1.1  christos 	kfail "continue to marker 2"
    735   1.1  christos 
    736   1.1  christos 
    737   1.1  christos XFAIL
    738   1.1  christos 
    739   1.1  christos Short for "expected failure", this indicates a known problem with the
    740   1.1  christos environment.  This could include limitations of the operating system,
    741   1.1  christos compiler version, and other components.
    742   1.1  christos 
    743   1.1  christos This example from gdb.base/attach-pie-misread.exp is a sanity check
    744   1.1  christos for the target environment:
    745   1.1  christos 
    746   1.1  christos 	# On x86_64 it is commonly about 4MB.
    747   1.1  christos 	if {$stub_size > 25000000} {
    748   1.1  christos 	    xfail "stub size $stub_size is too large"
    749   1.1  christos 	    return
    750   1.1  christos 	}
    751   1.1  christos 
    752   1.1  christos You should provide bug report number for the failing component of the
    753   1.1  christos environment, if such bug report is available, as with this example
    754   1.1  christos referring to a GCC problem:
    755   1.1  christos 
    756   1.1  christos 	  if {[test_compiler_info {gcc-[0-3]-*}]
    757   1.1  christos 	      || [test_compiler_info {gcc-4-[0-5]-*}]} {
    758   1.1  christos 	      setup_xfail "gcc/46955" *-*-*
    759   1.1  christos 	  }
    760   1.1  christos 	  gdb_test "python print ttype.template_argument(2)" "&C::c"
    761   1.1  christos 
    762   1.1  christos Note that it is also acceptable, and often preferable, to avoid
    763   1.1  christos running the test at all.  This is the better option if the limitation
    764   1.1  christos is intrinsic to the environment, rather than a bug expected to be
    765   1.1  christos fixed in the near future.
    766   1.8  christos 
    767   1.8  christos Local vs Remote vs Native
    768   1.8  christos *************************
    769   1.8  christos 
    770   1.8  christos It's unfortunately easy to get confused in the testsuite about what's
    771   1.8  christos native and what's not, what's remote and what's not.  The confusion is
    772   1.8  christos caused by the overlap in vocabulary between DejaGnu and GDB.
    773   1.8  christos 
    774   1.8  christos From a DejaGnu point of view:
    775   1.8  christos 
    776   1.8  christos  - native: the host or target board is considered native if the its
    777   1.8  christos    triplet is the same as the build system's triplet,
    778   1.8  christos 
    779   1.8  christos  - remote: the host or target board is considered remote if it's
    780   1.8  christos    running on a different machine, and thus require ssh, for example,
    781   1.8  christos    to run commands, versus simply running commands directly.
    782   1.8  christos 
    783   1.8  christos Note that they are not mutually exclusive, as you can have a remote
    784   1.8  christos machine that has the same triplet as the build machine.
    785   1.8  christos 
    786   1.8  christos From a GDB point of view:
    787   1.8  christos 
    788   1.8  christos  - native: when GDB uses system calls such as ptrace to interact
    789   1.8  christos    directly with processes on the same system its running on,
    790   1.8  christos 
    791   1.8  christos  - remote: when GDB speaks the RSP (Remote Serial Protocol) with
    792   1.8  christos    another program doing the ptrace stuff.
    793   1.8  christos 
    794   1.8  christos Note that they are mutually exclusive.  An inferior can only be either
    795   1.8  christos debugged with the native target, or with the remote target a specific
    796   1.8  christos time.
    797   1.8  christos 
    798   1.8  christos That means that there are cases where the target is not remote for
    799   1.8  christos DejaGnu, but is remote for GDB (e.g. running GDBserver on the same
    800   1.8  christos machine).
    801   1.8  christos 
    802   1.8  christos You can also have a remote target for DejaGnu, but native for GDB
    803   1.8  christos (e.g.  building on x86 a GDB that runs on ARM and running the
    804   1.8  christos testsuite with a remote host).
    805   1.8  christos 
    806   1.8  christos Therefore, care must be taken to check for the right kind of remote.
    807   1.8  christos Use [is_remote target] to check whether the DejaGnu target board is
    808   1.8  christos remote.  When what you really want to know is whether GDB is using the
    809   1.8  christos remote protocol, because feature X is only available when GDB debugs
    810   1.8  christos natively, check gdb_protocol instead.
    811