Home | History | Annotate | only in /src/external/mpl/bind/dist/bin/tests/system
Up to higher level directory
NameDateSize
_common/17-Sep-2026
acl/25-Feb-2026
additional/17-Sep-2026
addzone/08-Apr-2026
allow_query/19-Jun-2026
allow_query_on/17-Sep-2026
ans.py08-Apr-20261.7K
auth/08-Apr-2026
authsock.pl17-Sep-20262.2K
autosign/25-Feb-2026
bailiwick/08-Apr-2026
builtin/25-Feb-2026
cacheclean/17-Sep-2026
camp/25-Feb-2026
cap_glues/19-Jun-2026
case/25-Feb-2026
catz/20-May-2026
cdnxdomain/29-Aug-2026
cds/29-Aug-2026
chain/29-Aug-2026
checkconf/17-Sep-2026
checkconf_keys/08-Apr-2026
checkds/17-Sep-2026
checknames/25-Feb-2026
checkzone/17-Sep-2026
cipher_suites/29-Aug-2026
class/20-May-2026
cname_dname_negcache/29-Aug-2026
cname_recursion/17-Sep-2026
conf.sh29-Aug-202613.2K
conftest.py29-Aug-202624.2K
convert_junit_to_trs.py07-Apr-20261.8K
COOKBOOK.md17-Sep-202610.5K
cookie/29-Aug-2026
cpu/29-Aug-2026
custom-test-driver08-Apr-20265.1K
database/08-Apr-2026
dialup/19-Jun-2026
digcomp.pl23-Sep-20223.5K
digdelv/Today
dispatch/19-Jun-2026
ditch.pl22-Sep-20242.3K
dlzexternal/29-Aug-2026
dns64/17-Sep-2026
dns64_dname/19-Jun-2026
dns_import_checker.py07-Apr-20266.1K
dnssec/17-Sep-2026
dnssec_bypass/29-Aug-2026
dnssec_cname_response/29-Aug-2026
dnssec_nsec3/29-Aug-2026
dnssec_parent_rrsig/29-Aug-2026
dnssec_py/17-Sep-2026
dnssec_wildcard/29-Aug-2026
dnstap/29-Aug-2026
doth/29-Aug-2026
dsdigest/08-Apr-2026
dyndb/29-Aug-2026
ecdsa/08-Apr-2026
eddsa/29-Aug-2026
ednscompliance/25-Feb-2026
emptyzones/25-Feb-2026
enginepkcs11/29-Aug-2026
expiredglue/08-Apr-2026
feature-test.c20-Jun-20266K
fetchlimit/29-Aug-2026
filters/17-Sep-2026
formerr/29-Aug-2026
forward/29-Aug-2026
fromhex.pl23-Sep-20221K
fwdfirst/29-Aug-2026
genzone.sh29-Jan-202612.8K
geoip2/17-Sep-2026
get_core_dumps.sh21-Feb-20242.5K
glue/08-Apr-2026
hooks/08-Apr-2026
host/25-Feb-2026
idna/25-Feb-2026
ifconfig.sh.in29-Jan-20267.4K
include_multiplecfg/08-Apr-2026
inline/29-Aug-2026
integrity/25-Feb-2026
isctest/17-Sep-2026
ixfr/29-Aug-2026
ixfr_nonminimal/29-Aug-2026
journal/25-Feb-2026
kasp/Today
keepalive/08-Apr-2026
keyfromlabel/08-Apr-2026
ksr/29-Aug-2026
legacy/25-Feb-2026
limits/08-Apr-2026
logfileconfig/25-Feb-2026
Makefile.am17-Sep-20263.5K
Makefile.in17-Sep-202697.8K
makejournal.c29-Jan-20263.7K
masterfile/25-Feb-2026
masterformat/25-Feb-2026
metadata/25-Feb-2026
migrate2kasp/29-Aug-2026
mirror/25-Feb-2026
mirror_root_zone/19-Jun-2026
mismatchtcp/19-Jun-2026
mkeys/25-Feb-2026
multisigner/17-Sep-2026
names/08-Apr-2026
notify/08-Apr-2026
nsec/08-Apr-2026
nsec3/29-Aug-2026
nsec3_impersonation/29-Aug-2026
nsec_grandparent/17-Sep-2026
nsec_ixfr/20-May-2026
nsec_piggyback/29-Aug-2026
nsec_synthesis/29-Aug-2026
nsec_wildcard_wrong_zone/17-Sep-2026
nslookup/25-Feb-2026
nsprocessinglimit/08-Apr-2026
nsupdate/17-Sep-2026
nta/29-Aug-2026
nzd2nzf/08-Apr-2026
optout/19-Jun-2026
org.isc.bind.system22-Sep-2024791
org.isc.bind.system.plist12-Aug-2018425
outofzone_zonecut/17-Sep-2026
packet.pl20-May-20264.3K
padding/25-Feb-2026
pending/25-Feb-2026
pipelined/29-Aug-2026
proxy/29-Aug-2026
pytest.ini08-Apr-2026997
qmin/08-Apr-2026
qpcache_rrsig_any/19-Jun-2026
query_source/08-Apr-2026
randomizens/08-Apr-2026
re_compile_checker.py08-Apr-20261.7K
README.md17-Sep-202629.5K
reclimit/29-Aug-2026
redirect/19-Jun-2026
requirements.txt17-Sep-2026162
resend_loop/20-May-2026
resolver/29-Aug-2026
rfc5011/08-Apr-2026
rndc/17-Sep-2026
rndc_confgen/17-Sep-2026
rollover/Today
rootkeysentinel/25-Feb-2026
rpz/17-Sep-2026
rpzextra/08-Apr-2026
rpzrecurse/29-Aug-2026
rrchecker/25-Feb-2026
rrl/25-Feb-2026
rrsetorder/25-Feb-2026
rsabigexponent/Today
run.gdb24-May-202025
run.sh17-Jul-2025834
runtime/17-Sep-2026
selfpointedglue/29-Aug-2026
selftest/17-Sep-2026
serve_stale/17-Sep-2026
sfcache/25-Feb-2026
sfcache_cname/29-Aug-2026
shutdown/08-Apr-2026
sig0/08-Apr-2026
sig0_https/17-Sep-2026
smartsign/25-Feb-2026
sortlist/08-Apr-2026
spf/25-Feb-2026
srtt/20-May-2026
ssumaxtype/08-Apr-2026
ssutoctou/08-Apr-2026
start.pl20-May-202610.4K
start.sh.in21-Feb-2024546
staticstub/25-Feb-2026
statistics/29-Aug-2026
statschannel/29-Aug-2026
stop.pl26-Jan-20255.4K
stop.sh.in21-Feb-2024569
stress/Today
stub/08-Apr-2026
svcb_alias/17-Sep-2026
synthfromdnssec/25-Feb-2026
tcp/Today
testsock.pl21-Feb-20241.3K
testsock6.pl22-Sep-2024759
timeouts/29-Aug-2026
tkey/17-Sep-2026
tkeyleak/29-Aug-2026
tools/08-Apr-2026
transport_acl/08-Apr-2026
transport_change/29-Aug-2026
tsig/17-Sep-2026
tsiggss/17-Sep-2026
ttl/25-Feb-2026
unknown/20-May-2026
upforwd/29-Aug-2026
verify/17-Sep-2026
views/25-Feb-2026
vulture_ignore_list.py08-Apr-2026493
wildcard/29-Aug-2026
xfer/17-Sep-2026
xfer_servers_list/19-Jun-2026
xferquota/19-Jun-2026
zero/29-Aug-2026
zonechecks/17-Sep-2026

README.md

      1 <!--
      2 Copyright (C) Internet Systems Consortium, Inc. ("ISC")
      3 
      4 SPDX-License-Identifier: MPL-2.0
      5 
      6 This Source Code Form is subject to the terms of the Mozilla Public
      7 License, v. 2.0.  If a copy of the MPL was not distributed with this
      8 file, you can obtain one at https://mozilla.org/MPL/2.0/.
      9 
     10 See the COPYRIGHT file distributed with this work for additional
     11 information regarding copyright ownership.
     12 -->
     13 
     14 # BIND9 System Test Framework
     15 
     16 This directory holds test environments for running bind9 system tests involving
     17 multiple name servers.
     18 
     19 Each system test directory holds a set of test scripts and configuration files
     20 to test different parts of BIND.  The directories are named for the aspect of
     21 BIND they test, for example:
     22 
     23 ```
     24 dnssec/       DNSSEC tests
     25 forward/      Forwarding tests
     26 glue/         Glue handling tests
     27 ```
     28 
     29 etc.
     30 
     31 A system test directory name must start with an alphabetic character and may
     32 only contain alphanumeric characters and underscores.  Use underscore as the
     33 word separator; hyphens are not allowed  they are reserved for the temporary
     34 directories and symlinks the test runner creates.
     35 
     36 Typically each set of tests sets up 2-5 name servers and then performs one or
     37 more tests against them.  Within the test subdirectory, each name server has a
     38 separate subdirectory containing its configuration data.  These subdirectories
     39 are named "nsN" or "ansN" (where N is a number between 1 and 11, e.g. ns1,
     40 ans2 etc.)
     41 
     42 The tests are completely self-contained and do not require access to the real
     43 DNS.  Generally, one of the test servers (usually ns1) is set up as a root
     44 nameserver and is listed in the hints file of the others.
     45 
     46 For task-oriented recipes (adding a new test directory, writing a regression
     47 reproducer, mocking a misbehaving server, setting up zones), see
     48 [the cookbook](COOKBOOK.md).
     49 
     50 
     51 ## Running the Tests
     52 
     53 ### Building BIND
     54 
     55 The system tests run the binaries from the build tree, so BIND must be built
     56 first.  The test-only binaries and plugins the tests need are built as part
     57 of the regular build:
     58 
     59 ```sh
     60 autoreconf -fi
     61 ./configure
     62 make -j
     63 ```
     64 
     65 ### Prerequisites
     66 
     67 To run system tests, make sure you have the following dependencies installed:
     68 
     69 - python3 (3.10 and newer)
     70 - pytest (7.0 and newer)
     71 - pytest-xdist
     72 - perl (still needed by the test runner internals; some legacy tests
     73   additionally need the Net::DNS module and are skipped when it is missing)
     74 
     75 The full list of required and optional python packages can be found in
     76 [requirements.txt](requirements.txt) (it can be installed with
     77 `pip3 install -r requirements.txt`).
     78 
     79 ### Network Setup
     80 
     81 To enable all servers to run on the same machine, they bind to separate virtual
     82 IP addresses on the loopback interface.  ns1 runs on 10.53.0.1, ns2 on
     83 10.53.0.2, etc.  Before running any tests, you must set up these addresses by
     84 running the command
     85 
     86 ```sh
     87 sh ifconfig.sh up
     88 ```
     89 
     90 as root.  The interfaces can be removed by executing the command:
     91 
     92 ```sh
     93 sh ifconfig.sh down
     94 ```
     95 
     96 ... also as root.
     97 
     98 The servers use unprivileged ports (above 1024) instead of the usual port 53,
     99 so they can be run without root privileges once the interfaces have been set
    100 up.
    101 
    102 **Note for MacOS Users**
    103 
    104 If you wish to make the interfaces survive across reboots, copy
    105 org.isc.bind.system and org.isc.bind.system.plist to /Library/LaunchDaemons
    106 then run
    107 
    108 ```sh
    109 launchctl load /Library/LaunchDaemons/org.isc.bind.system.plist
    110 ```
    111 
    112 ... as root.
    113 
    114 ### Running All the System Tests
    115 
    116 Issue a plain `pytest` command in this directory to execute all tests
    117 sequentially.  To execute them in parallel instead, run:
    118 
    119 ```sh
    120 pytest -n <number-of-workers>
    121 ```
    122 
    123 Parallel execution requires pytest-xdist; `-n auto` uses one worker per CPU.
    124 
    125 Alternately, using the make command is also supported:
    126 
    127 ```sh
    128 make [-j numproc] test
    129 ```
    130 
    131 ### Running a Single Test
    132 
    133 To run all test modules in a single system test directory, pass the directory
    134 name to pytest:
    135 
    136 ```sh
    137 pytest dns64
    138 ```
    139 
    140 The utility script `./run.sh dns64` does the same thing.
    141 
    142 To narrow the run down further, prefer pytest node IDs over `-k` matching 
    143 they are exact:
    144 
    145 ```sh
    146 pytest dnssec_py/tests_mixed_ds.py
    147 pytest doth/tests_sslyze.py::test_sslyze_dot
    148 ```
    149 
    150 Parametrized tests have the parameter ID in brackets, so a single case of a
    151 parametrized test can be selected as:
    152 
    153 ```sh
    154 pytest "dnssec_py/tests_nsec3_answer.py::test_nodata[ns2]"
    155 ```
    156 
    157 The `-k` option selects tests by pattern matching:
    158 
    159 ```sh
    160 pytest -k <test-name-or-pattern>
    161 ```
    162 
    163 Beware that a `-k` pattern might pick up more tests than intended.  Use the
    164 `--collect-only` option to check the list of tests which match your `-k`
    165 pattern.
    166 
    167 ### rr
    168 
    169 When running system tests, named can be run under the rr tool. rr records a
    170 trace to the $system_test/nsX/named-Y/ directory, which can be later used to
    171 replay named. To enable this, run pytest with the USE_RR environment variable
    172 set.
    173 
    174 ### Test Artifacts
    175 
    176 Each test module is executed inside a unique temporary directory which contains
    177 all the artifacts from the test run. If the tests succeed, they are deleted by
    178 default. To override this behaviour, pass `--noclean` to pytest.
    179 
    180 The directory name starts with the system test name, followed by `-tmp-XXXXXX`,
    181 i.e. `dns64-tmp-r07vei9s` for `dns64` test run. Since this name changes each
    182 run, a convenience symlink that has a stable name is also created. It points to
    183 the latest test artifacts directory and has a form of `dns64-sh_dns64`
    184 (depending on the particular test module).
    185 
    186 To clean up the temporary directories and symlinks, run `make clean-local` in
    187 the system test directory.
    188 
    189 The following test artifacts are typically available:
    190 
    191 - pytest.log.txt: main log file with test output
    192 - files generated by the test itself, e.g. output from "dig" and "rndc"
    193 - files produced by named, other tools or helper scripts
    194 
    195 
    196 ## Writing System Tests
    197 
    198 ### File Overview
    199 
    200 Tests are organized into system test directories which may hold one or more
    201 test modules (python files). Each module may have multiple test cases. The
    202 system test directories may contain the following standard files:
    203 
    204 - `tests_*.py`: These python files are picked up by pytest as modules. If they
    205   contain any test functions, they're added to the test suite.
    206 
    207 - `*.j2`: Jinja2 templates, rendered automatically during test setup (see
    208   [Templates](#templates) below).
    209 
    210 - `ns<N>`: These subdirectories contain test name servers that can be queried
    211   or can interact with each other. The value of N indicates the address the
    212   server listens on: for example, ns2 listens on 10.53.0.2, and ns4 on
    213   10.53.0.4. All test servers use an unprivileged port, so they don't need to
    214   run as root. These servers log at the highest debug level and the log is
    215   captured in the file "named.run".
    216 
    217 - `ans<N>`: Like ns<N>, but these are mock name servers implemented in python
    218   (`ans.py`), usually with the `isctest.asyncserver` module.  They are
    219   generally programmed to misbehave in ways named would not, so as to exercise
    220   named's ability to interoperate with badly behaved name servers.  A few
    221   legacy mock servers are still implemented in perl (`ans.pl`); don't write
    222   new ones.
    223 
    224 The following files appear in test directories that have not yet been fully
    225 ported to python; do not add them to new tests:
    226 
    227 - `tests.sh`: Legacy shell-based tests, run via a `tests_sh_*.py` glue module.
    228 
    229 - `setup.sh`: Legacy shell test setup.  New tests use templates and a
    230   `bootstrap()` function instead.
    231 
    232 ### Module Scope
    233 
    234 A module is a python file which contains test functions. Every system
    235 test directory may contain multiple modules (i.e. tests_*.py files).
    236 
    237 The server setup/teardown is performed for each module. Bundling test cases
    238 together inside a single module may save some resources. However, test cases
    239 inside a single module can't be executed in parallel.
    240 
    241 It is possible to execute different modules defined within a single system test
    242 directory in parallel. This is possible thanks to executing the tests inside a
    243 temporary directory and proper port assignment to ensure there won't be any
    244 conflicts.
    245 
    246 ### Port Usage
    247 
    248 In order for the tests to run in parallel, each test requires a unique set of
    249 ports. This is ensured by the pytest runner, which assigns a unique set of
    250 ports to each test module.
    251 
    252 Inside the python tests, it is possible to use fixtures like `named_port` to
    253 get the assigned port numbers. They're also set as environment variables.
    254 These include:
    255 
    256 - `PORT`: used as the basic dns port
    257 - `TLSPORT`: used as the port for DNS-over-TLS
    258 - `HTTPPORT`, `HTTPSPORT`: used as the ports for DNS-over-HTTP(S)
    259 - `CONTROLPORT`: used as the RNDC control port
    260 - `EXTRAPORT1` through `EXTRAPORT8`: additional ports that can be used as needed
    261 
    262 ### Templates
    263 
    264 Configuration files which need values that are only known at test run time 
    265 ports, default crypto algorithms, conditional sections  are written as jinja2
    266 templates with a `.j2` extension.  During test setup, the pytest runner
    267 renders every `*.j2` file in the test directory and strips the extension:
    268 `ns1/named.conf.j2` becomes `ns1/named.conf`.
    269 
    270 Inside a template, all the runner's environment variables are available with
    271 `@...@` delimiters, e.g.:
    272 
    273 ```jinja
    274 options {
    275 	{% include_indented "_common/options.conf.j2" %}
    276 	tls-port @TLSPORT@;
    277 };
    278 ```
    279 
    280 Standard jinja2 block syntax (`{% if %}`, `{% for %}`, ) can be used for
    281 conditional or repeated sections.
    282 
    283 Custom template variables come from an optional module-level `bootstrap()`
    284 function.  When a test module defines one, the runner calls it before
    285 rendering the templates and passes the returned dict to the template engine:
    286 
    287 ```python
    288 def bootstrap():
    289     return {"valid": True}
    290 ```
    291 
    292 Templates using custom variables must always provide defaults, so that the
    293 file also renders when no value is supplied (e.g. when another module in the
    294 same directory has no `bootstrap()`):
    295 
    296 ```jinja
    297 {% set valid = valid | default(False) %}
    298 ```
    299 
    300 `bootstrap()` is also the place where a module generates test data that has
    301 to exist before the servers start  typically zone files and DNSSEC keys (see
    302 [the cookbook](COOKBOOK.md) for a complete example).
    303 
    304 Templates can also be re-rendered while the test is running, using the
    305 `templates` fixture, e.g. to change a server's config before reloading it:
    306 
    307 ```python
    308 def test_reload(ns1, templates):
    309     templates.render("ns1/named.conf", {"valid": True})
    310     ns1.reconfigure()
    311 ```
    312 
    313 If you don't need the file to be auto-templated during test setup, use the
    314 extension `.j2.manual` instead; such templates are only rendered when the test
    315 calls `templates.render()` explicitly, and no defaults are needed.
    316 
    317 ### Fixtures and Helpers
    318 
    319 Fixtures defined in `conftest.py` provide the test context:
    320 
    321 - `servers` is a dictionary of all started `isctest.instance.NamedInstance`
    322   servers, keyed by directory name; the shortcut fixtures `ns1` through `ns11`
    323   return the corresponding instance directly. A `NamedInstance` is the
    324   interface for driving a server: `ns1.rndc("...")`, `ns1.reconfigure()`,
    325   `ns1.nsupdate(...)`, `ns1.watch_log_from_here()`, `ns1.ip`, ...
    326 - `templates` renders jinja2 templates at runtime (see above).
    327 - `named_port`, `named_tlsport`, `control_port`, ... return the assigned port
    328   numbers.
    329 - `system_test_dir` is the temporary directory the module runs in.
    330 
    331 The `isctest` package provides the helper library; the modules most tests
    332 need are `isctest.query` (send DNS queries), `isctest.check` (assert on
    333 responses), `isctest.zone` (zone and key setup), `isctest.kasp` (DNSSEC
    334 key state checks), `isctest.asyncserver` (mock servers) and `isctest.log`
    335 (logging and log watchers).
    336 
    337 Pytest marks control test collection and setup:
    338 
    339 - `@pytest.mark.extra_artifacts([...])` declares the files (globs) the test is
    340   expected to leave behind in addition to the common ones; undeclared
    341   leftovers fail the run.
    342 - `@pytest.mark.requires_zones_loaded("ns1", ...)` delays the test until the
    343   listed servers have loaded all zones.
    344 - `isctest.mark` has skip-unless conditions for environment prerequisites,
    345   e.g. `isctest.mark.with_dnstap`, `isctest.mark.with_lmdb`,
    346   `isctest.mark.live_internet_test`.
    347 
    348 ### Parametrization
    349 
    350 Use `pytest.mark.parametrize` to run one test function over several inputs
    351 instead of copy-pasting test cases or looping inside one test function  each
    352 case is reported (and can be re-run) individually:
    353 
    354 ```python
    355 @pytest.mark.parametrize(
    356     "qname,rdtype",
    357     [
    358         ("exists.example.", "A"),
    359         ("exists.example.", "TXT"),
    360         ("other.example.", "A"),
    361     ],
    362 )
    363 def test_answers(qname, rdtype, ns1):
    364     msg = isctest.query.create(qname, rdtype)
    365     response = isctest.query.udp(msg, ns1.ip)
    366     isctest.check.noerror(response)
    367 ```
    368 
    369 This creates the test cases `test_answers[exists.example.-A]` etc., which can
    370 be passed to pytest as node IDs.
    371 
    372 ### Logging
    373 
    374 Each module has a separate log which will be saved as pytest.log.txt in the
    375 temporary directory in which the test is executed. This log includes messages
    376 for this module setup/teardown as well as any logging from the tests. Logging
    377 level DEBUG and above will be present in this log.
    378 
    379 Use `isctest.log` for test output (`isctest.log.info("...")` etc.); in
    380 general, any log messages using INFO or above will also be printed out during
    381 pytest execution. In CI, the pytest output is also saved to pytest.out.txt in
    382 the bin/tests/system directory.
    383 
    384 ### Adding a Test to the System Test Suite
    385 
    386 Once a test has been created it will be automatically picked up by the pytest
    387 runner if it upholds the convention expected by pytest (especially when it
    388 comes to naming files and test functions).  A new system test directory also
    389 needs to be added to `TESTS` in `Makefile.am`, in order to be included in
    390 `make check` runs.
    391 
    392 
    393 ### Writing a regression reproducer
    394 
    395 The goal: turn "issue #NNNN" into a failing test with minimal ceremony.
    396 
    397 1. **Decide the server topology.** Most reproducers need one of:
    398    - a single authoritative `named` (answer content bugs)  the skeleton
    399      recipe above;
    400    - a resolver plus a mock server that misbehaves (resolver bugs)  see
    401      the mock server recipe below;
    402    - signed zones and a validating resolver (DNSSEC bugs)  see the zone
    403      setup recipe below.
    404 
    405 2. **Find the closest existing test and copy its shape.**  Good exemplars:
    406    `dispatch` (resolver + python mock server), `dnssec_py` (signed zones,
    407    validator, multiple modules sharing one server set), `nsec3` (multi-module
    408    family), `kasp`/`rollover_*` (key management state machines).
    409 
    410 3. **Decide where the test lives.**  If an existing directory already has the
    411    server set you need, add a new `tests_*.py` module there; otherwise create
    412    a new directory.  Each module gets its own temporary directory, port
    413    range, and parallel slot, so you are not entangled with the other modules.
    414    Test functions *within* a module, however, run in file order against the
    415    same live servers: a new test inherits whatever state the tests above it
    416    left behind (cache contents, dynamic updates) and can disturb the tests
    417    below it.
    418 
    419 4. **Write the test to fail first.**  Run it against an unfixed build and
    420    make sure it fails for the reason the issue describes  `ns*/named.run`
    421    in the kept temporary directory is the place to verify that.  Then apply
    422    the fix and watch it pass.
    423 
    424 
    425 ### Mock a misbehaving server
    426 
    427 When a test needs a server that answers in ways named never would (bogus
    428 glue, truncation, dropped queries, malformed records), add an `ansN`
    429 subdirectory containing an `ans.py` script based on `isctest.asyncserver`.
    430 The runner starts it automatically on 10.53.0.N, logging to `ans.run`.
    431 
    432 Implementing a custom `ansN` server happens in two phases:
    433 
    434   - define all static DNS data that the server needs to serve (if any) in `*.db`
    435     files, like you would for a regular `named` instance,
    436 
    437   - implement any non-standard behavior (modifying zone-based responses or
    438     generating responses from scratch) by defining a response handler class,
    439     scoping it to the QNAMEs/QTYPEs/domains it owns, and installing it into an
    440     `AsyncDnsServer`.
    441 
    442 Most importantly, avoid the temptation to define all DNS responses that a given
    443 `ansN` server needs to serve using just dnspython APIs; zone files are much
    444 easier to follow for static DNS data.  Splitting up static DNS data and custom
    445 behavior also makes it easier to follow the idea behind each test.
    446 
    447 The most commonly subclassed handler classes are (ordered by descending
    448 specificity):
    449 
    450   - `QnameQtypeHandler`
    451   - `QnameHandler`
    452   - `DomainHandler`
    453 
    454 These handler classes require certain properties (e.g. `qnames`, `qtypes`,
    455 `domains`) to be defined by their subclasses.  These properties define the set
    456 of queries that a given handler should be used for.  Please see
    457 `isctest/asyncserver.py` for up-to-date information on available handler classes
    458 and existing `ans.py` files for how they can be used in practice.  Consult the
    459 log files (`ans.run`) in case a query is not matched by its intended handler.
    460 
    461 **NOTE:** For readability (of both code and logs), defining separate handler
    462 classes for distinct queries is strongly preferred over using a single handler
    463 containing an `if`/`elif`/`else` chain.
    464 
    465 **NOTE:** If you find yourself implementing an `__init__()` method in your
    466 handler subclass, it often indicates that you're approaching the problem at hand
    467 from the wrong side; contact QA for guidance in such a case.
    468 
    469 When a query is matched to a handler, the latter is expected to yield a response
    470 action through its `get_responses()` method, an async generator that inspects
    471 the query context and decides how the server should react:
    472 
    473 ```python
    474 from collections.abc import AsyncGenerator
    475 
    476 import dns.flags
    477 
    478 from isctest.asyncserver import (
    479     AsyncDnsServer,
    480     DnsResponseSend,
    481     DomainHandler,
    482     QueryContext,
    483     ResponseAction,
    484 )
    485 
    486 
    487 class TruncateHandler(DomainHandler):
    488     """Answer everything under broken.example. with TC=1."""
    489 
    490     domains = ["broken.example."]
    491 
    492     async def get_responses(
    493         self, qctx: QueryContext
    494     ) -> AsyncGenerator[ResponseAction, None]:
    495         qctx.response.flags |= dns.flags.TC
    496         yield DnsResponseSend(qctx.response)
    497 
    498 
    499 def main() -> None:
    500     server = AsyncDnsServer()
    501     server.install_response_handler(TruncateHandler())
    502     server.run()
    503 
    504 
    505 if __name__ == "__main__":
    506     main()
    507 ```
    508 
    509 The available response actions are `DnsResponseSend` (optionally with a
    510 `delay`), `ResponseDrop` (don't answer at all), `BytesResponseSend` (raw
    511 bytes, for malformed packets) and `CloseConnection` (TCP).  Queries that no
    512 handler matches are answered from zone data  `AsyncDnsServer` loads every
    513 `*.db` zone file found in the `ansN` directory at startup  or with the
    514 server's default rcode (REFUSED unless configured otherwise).
    515 
    516 **NOTE:** For returning static responses, subclassing `StaticResponseHandler` is
    517 strongly recommended instead of implementing the `get_responses()` generator
    518 manually; see `resolver/ans3/ans.py` for practical examples.
    519 
    520 **NOTE:** Calling `yield` does **NOT** make `get_responses()` return!  This is
    521 by design: `get_responses()` can yield multiple DNS messages in response to a
    522 single query, so that it can also handle AXFR/IXFR queries, among others.  Be
    523 careful not to unintentionally cause multiple DNS messages to be returned for a
    524 single query.  If your handler's `get_responses()` method contains multiple
    525 `yield` statements, it might be a sign that it needs to be refactored into
    526 multiple separate handlers.
    527 
    528 If multiple `ansN` instances used in a given system test need to share common
    529 logic, extract that logic into a `<test-name>_ans.py` module in the system test
    530 directory.  See the `qmin` system test for a practical example.
    531 
    532 If multiple system tests would benefit from sharing some common logic, consider
    533 submitting a merge request adding that logic to `isctest/asyncserver.py` itself.
    534 
    535 To the extent possible, try to keep each `ans.py` file limited in length and
    536 scope.  Look at existing `ans.py` files to see what is meant by that.  If the
    537 response generation logic required for reproducing a given bug is particularly
    538 complex, consider dedicating the entire `ans.py` file just to that logic instead
    539 of appending it to an existing one; `ansN` instances are cheap to spawn and run
    540 compared to regular `named` instances.  If the number of `ansN` instances used
    541 in a given system test is becoming unwieldy, it usually indicates the need to
    542 start adding/moving code to a new system test directory.
    543 
    544 In some rare cases, it may be useful to reuse a common set of `nsN` server
    545 instances to reproduce a whole class of related issues, triggering which relies
    546 on some non-standard behavior and therefore needs a custom `ansN` server to be
    547 implemented.  If the logic necessary for reproducing each of these issues is
    548 complex and the amount of those issues makes it impractical to add a separate
    549 `ansN` server for each issue (as recommended in the previous paragraph), it is
    550 acceptable to split up the test logic for each issue into separate `ans_*.py`
    551 modules inside a single `ansN` directory and reduce `ans.py` itself to a loader
    552 that imports and installs handlers defined in those separate modules:
    553 
    554 ```python
    555 from mytest.ans1 import ans_some_bug, ans_some_other_bug
    556 from isctest.asyncserver import AsyncDnsServer
    557 
    558 
    559 def main() -> None:
    560     server = AsyncDnsServer()
    561     server.install_response_handler(ans_some_bug.SomeBugHandler())
    562     server.install_response_handler(ans_some_other_bug.SomeOtherBugHandler())
    563     server.run()
    564 
    565 
    566 if __name__ == "__main__":
    567     main()
    568 ```
    569 
    570 However, in such a case it is particularly important to ensure consistency
    571 between the names of all the Python files related to a given issue - otherwise,
    572 chaos ensues.  Furthermore, avoid using cryptic file names (e.g. numeric bug
    573 identifiers).  The recommended naming scheme is:
    574 
    575 ```
    576 mytest/
    577  ans1
    578   ans.py
    579   ans_some_bug.py
    580   ans_some_other_bug.py
    581  ns2
    582   ...
    583  tests_some_bug.py
    584  tests_some_other_bug.py
    585 ```
    586 
    587 To point a resolver at the mock, delegate to it from the test's root zone
    588 (served by ns1) or list it as a forwarder; `dispatch` shows the
    589 delegation pattern end to end.
    590 
    591 The existing mock servers are the best reference.  To find them, grep for
    592 what you're about to use:
    593 `git grep -l isctest.asyncserver -- '*/ans*/ans.py'` lists every python
    594 mock, and a grep for the base class
    595 (`DomainHandler`, `QnameHandler`, `ConnectionHandler`) or the response
    596 action (`ResponseDrop`, `BytesResponseSend`, ...) you need usually turns
    597 up a test already doing something similar.  The full toolbox lives in
    598 `isctest/asyncserver.py` (query matching, TCP connection handling, TSIG
    599 keyrings).
    600 
    601 
    602 ## Nameservers
    603 
    604 As noted earlier, a system test will involve a number of nameservers.  These
    605 will be either instances of named, or mock servers, typically written in
    606 Python.
    607 
    608 For the former, the version of "named" being run is the one from the build
    609 tree set up by `./configure` (i.e. if the tests are run immediately after
    610 `make`, the version of "named" used is the one just built).  The
    611 configuration files, zone files etc. for these servers are located in
    612 subdirectories of the test directory named "nsN", where N is a small integer.
    613 The latter are special nameservers, mostly used for generating deliberately bad
    614 responses, located in subdirectories named "ansN" (again, N is an integer).
    615 In addition to configuration files, these directories should hold the
    616 appropriate script files as well.
    617 
    618 Note that the "N" for a particular test forms a single number space, e.g. if
    619 there is an "ns2" directory, there cannot be an "ans2" directory as well.
    620 Ideally, the directory numbers should start at 1 and work upwards.
    621 
    622 When tests are executed, pytest takes care of the test setup and teardown. It
    623 looks for any `nsN` and `ansN` directories in the system test directory and
    624 starts those servers.
    625 
    626 ### `named` Command-Line Options
    627 
    628 By default, `named` server is started with the following options:
    629 
    630 ```
    631 -c named.conf   Specifies the configuration file to use (so by implication,
    632                 each "nsN" nameserver's configuration file must be called
    633                 named.conf).
    634 
    635 -d 99           Sets the maximum debugging level.
    636 
    637 -D <name>       The "-D" option sets a string used to identify the
    638                 nameserver in a process listing.  In this case, the string
    639                 is the name of the subdirectory.
    640 
    641 -g              Runs the server in the foreground and logs everything to
    642                 stderr.
    643 
    644 -m record
    645                 Turns on these memory usage debugging flags.
    646 ```
    647 
    648 All output is sent to a file called `named.run` in the nameserver directory.
    649 
    650 The options used to start named can be altered. There are a couple ways of
    651 doing this. The runner checks the methods in a specific order: if a check
    652 succeeds, the options are set and any other specification is ignored.  In
    653 order, these are:
    654 
    655 1. Including a file called "named.args" in the "nsN" directory.  If present,
    656 the contents of the first non-commented, non-blank line of the file are used as
    657 the named command-line arguments.  The rest of the file is ignored.
    658 
    659 2. Tweaking the default command line arguments with "-T" options.  This flag is
    660 used to alter the behavior of BIND for testing and is not documented in the
    661 ARM.  The presence of a file called `named.<flag>` in the "nsN" directory adds
    662 `-T <flag>` to the default command line (the content of the file is irrelevant
    663 - it is only the presence that counts).  The recognized flags are:
    664 
    665 ```
    666 dropedns         Recognise EDNS options in messages, but drop messages
    667                  containing them.
    668 
    669 ednsformerr, ednsnotimp, ednsrefused
    670                  Answer EDNS queries with the given rcode, pretending to
    671                  be an old server that doesn't understand EDNS.
    672 
    673 cookiealwaysvalid
    674                  Accept any DNS cookie presented by a client.
    675 
    676 noaa             Never set the AA bit in an answer.
    677 
    678 noedns           Disable recognition of EDNS options in messages.
    679 
    680 nonearest        Omit the closest-encloser NSEC3 proof from negative
    681                  responses (except for DS queries).
    682 
    683 nosoa            Disable the addition of SOA records to negative
    684                  responses (or to the additional section if the response
    685                  is triggered by RPZ rewriting).
    686 
    687 maxudp512, maxudp1460
    688                  Set the maximum UDP size handled by named to 512/1460.
    689 
    690 tat=1, tat=3     Send trust-anchor-telemetry queries every N seconds.
    691 
    692 notcp            Disable TCP in "named".  Unlike the other flags, this
    693                  one is also applied when "named.args" is used.
    694 ```
    695 
    696 ### Running Nameservers Interactively
    697 
    698 In order to debug the nameservers, you can let pytest perform the nameserver
    699 setup and interact with the servers before the test starts, or even at specific
    700 points during the test, using the `--trace` option to drop you into pdb debugger
    701 which pauses the execution of the tests, while keeping the server state intact:
    702 
    703 ```sh
    704 pytest -k dns64 --trace
    705 ```
    706 
    707 
    708 ## Developer Notes
    709 
    710 ### Test discovery and collection
    711 
    712 There are two distinct types of system tests. The first is a legacy shell
    713 script tests.sh containing individual test cases executed sequentially and the
    714 success/failure is determined by return code. The second type is a regular
    715 pytest file which contains test functions.
    716 
    717 Dealing with the regular pytest files doesn't require any special consideration
    718 as long as the naming conventions are met. Discovering the tests.sh tests is
    719 more complicated.
    720 
    721 The chosen solution is to add a bit of glue for each system test. For every
    722 tests.sh, there is an accompanying tests_sh_*.py file that contains a test
    723 function which utilizes a custom run_tests_sh fixture to call the tests.sh
    724 script. Other solutions were tried and eventually rejected. While this
    725 introduces a bit of extra glue, it is the most portable, compatible and least
    726 complex solution.
    727 
    728 ### Compatibility with older pytest version
    729 
    730 The minimum supported versions of python and the required python packages are
    731 declared in [requirements.txt](requirements.txt) and in the
    732 `pytest_configure()` check in `conftest.py`.  When implementing new runner
    733 features, check feature support in the pytest and pytest-xdist versions
    734 available in the oldest distributions covered by CI first; we may need to add
    735 compat code to handle breaking upstream changes in either direction.
    736 
    737 ### Format of Shell Test Output
    738 
    739 Legacy shell-based tests have the following format of output:
    740 
    741 ```
    742 <letter>:<test-name>:<message> [(<number>)]
    743 ```
    744 
    745 e.g.
    746 
    747 ```
    748 I:catz:checking that dom1.example is not served by primary (1)
    749 ```
    750 
    751 The meanings of the fields are as follows:
    752 
    753 <letter>
    754 This indicates the type of message.  This is one of:
    755 
    756 ```
    757 S   Start of the test
    758 A   Start of test (retained for backwards compatibility)
    759 T   Start of test (retained for backwards compatibility)
    760 E   End of the test
    761 I   Information.  A test will typically output many of these messages
    762     during its run, indicating test progress.  Note that such a message may
    763     be of the form "I:testname:failed", indicating that a sub-test has
    764     failed.
    765 R   Result.  Each test will result in one such message, which is of the
    766     form:
    767 
    768             R:<test-tmpdir>:<result>
    769 
    770     where <result> is one of:
    771 
    772         PASS        The test passed
    773         FAIL        The test failed
    774         SKIPPED     The test was not run, usually because some
    775                     prerequisites required to run the test are missing.
    776 ```
    777 
    778 <test-tmpdir>
    779 This is the name of the temporary test directory from which the message
    780 emanated, which is also the name of the subdirectory holding the test files.
    781 
    782 <message>
    783 This is text output by the test during its execution.
    784 
    785 (<number>)
    786 If present, this will correlate with a file created by the test.  The tests
    787 execute commands and route the output of each command to a file.  The name of
    788 this file depends on the command and the test, but will usually be of the form:
    789 
    790 ```
    791 <command>.out.<suffix><number>
    792 ```
    793 
    794 e.g. nsupdate.out.test28, dig.out.q3.  This aids diagnosis of problems by
    795 allowing the output that caused the problem message to be identified.
    796