1 <!--
2 Copyright (C) Internet Systems Consortium, Inc. ("ISC")
3
4 SPDX-License-Identifier: MPL-2.0
5
6 This Source Code Form is subject to the terms of the Mozilla Public
7 License, v. 2.0. If a copy of the MPL was not distributed with this
8 file, you can obtain one at https://mozilla.org/MPL/2.0/.
9
10 See the COPYRIGHT file distributed with this work for additional
11 information regarding copyright ownership.
12 -->
13
14 # BIND9 System Test Framework
15
16 This directory holds test environments for running bind9 system tests involving
17 multiple name servers.
18
19 Each system test directory holds a set of test scripts and configuration files
20 to test different parts of BIND. The directories are named for the aspect of
21 BIND they test, for example:
22
23 ```
24 dnssec/ DNSSEC tests
25 forward/ Forwarding tests
26 glue/ Glue handling tests
27 ```
28
29 etc.
30
31 A system test directory name must start with an alphabetic character and may
32 only contain alphanumeric characters and underscores. Use underscore as the
33 word separator; hyphens are not allowed they are reserved for the temporary
34 directories and symlinks the test runner creates.
35
36 Typically each set of tests sets up 2-5 name servers and then performs one or
37 more tests against them. Within the test subdirectory, each name server has a
38 separate subdirectory containing its configuration data. These subdirectories
39 are named "nsN" or "ansN" (where N is a number between 1 and 11, e.g. ns1,
40 ans2 etc.)
41
42 The tests are completely self-contained and do not require access to the real
43 DNS. Generally, one of the test servers (usually ns1) is set up as a root
44 nameserver and is listed in the hints file of the others.
45
46 For task-oriented recipes (adding a new test directory, writing a regression
47 reproducer, mocking a misbehaving server, setting up zones), see
48 [the cookbook](COOKBOOK.md).
49
50
51 ## Running the Tests
52
53 ### Building BIND
54
55 The system tests run the binaries from the build tree, so BIND must be built
56 first. The test-only binaries and plugins the tests need are built as part
57 of the regular build:
58
59 ```sh
60 autoreconf -fi
61 ./configure
62 make -j
63 ```
64
65 ### Prerequisites
66
67 To run system tests, make sure you have the following dependencies installed:
68
69 - python3 (3.10 and newer)
70 - pytest (7.0 and newer)
71 - pytest-xdist
72 - perl (still needed by the test runner internals; some legacy tests
73 additionally need the Net::DNS module and are skipped when it is missing)
74
75 The full list of required and optional python packages can be found in
76 [requirements.txt](requirements.txt) (it can be installed with
77 `pip3 install -r requirements.txt`).
78
79 ### Network Setup
80
81 To enable all servers to run on the same machine, they bind to separate virtual
82 IP addresses on the loopback interface. ns1 runs on 10.53.0.1, ns2 on
83 10.53.0.2, etc. Before running any tests, you must set up these addresses by
84 running the command
85
86 ```sh
87 sh ifconfig.sh up
88 ```
89
90 as root. The interfaces can be removed by executing the command:
91
92 ```sh
93 sh ifconfig.sh down
94 ```
95
96 ... also as root.
97
98 The servers use unprivileged ports (above 1024) instead of the usual port 53,
99 so they can be run without root privileges once the interfaces have been set
100 up.
101
102 **Note for MacOS Users**
103
104 If you wish to make the interfaces survive across reboots, copy
105 org.isc.bind.system and org.isc.bind.system.plist to /Library/LaunchDaemons
106 then run
107
108 ```sh
109 launchctl load /Library/LaunchDaemons/org.isc.bind.system.plist
110 ```
111
112 ... as root.
113
114 ### Running All the System Tests
115
116 Issue a plain `pytest` command in this directory to execute all tests
117 sequentially. To execute them in parallel instead, run:
118
119 ```sh
120 pytest -n <number-of-workers>
121 ```
122
123 Parallel execution requires pytest-xdist; `-n auto` uses one worker per CPU.
124
125 Alternately, using the make command is also supported:
126
127 ```sh
128 make [-j numproc] test
129 ```
130
131 ### Running a Single Test
132
133 To run all test modules in a single system test directory, pass the directory
134 name to pytest:
135
136 ```sh
137 pytest dns64
138 ```
139
140 The utility script `./run.sh dns64` does the same thing.
141
142 To narrow the run down further, prefer pytest node IDs over `-k` matching
143 they are exact:
144
145 ```sh
146 pytest dnssec_py/tests_mixed_ds.py
147 pytest doth/tests_sslyze.py::test_sslyze_dot
148 ```
149
150 Parametrized tests have the parameter ID in brackets, so a single case of a
151 parametrized test can be selected as:
152
153 ```sh
154 pytest "dnssec_py/tests_nsec3_answer.py::test_nodata[ns2]"
155 ```
156
157 The `-k` option selects tests by pattern matching:
158
159 ```sh
160 pytest -k <test-name-or-pattern>
161 ```
162
163 Beware that a `-k` pattern might pick up more tests than intended. Use the
164 `--collect-only` option to check the list of tests which match your `-k`
165 pattern.
166
167 ### rr
168
169 When running system tests, named can be run under the rr tool. rr records a
170 trace to the $system_test/nsX/named-Y/ directory, which can be later used to
171 replay named. To enable this, run pytest with the USE_RR environment variable
172 set.
173
174 ### Test Artifacts
175
176 Each test module is executed inside a unique temporary directory which contains
177 all the artifacts from the test run. If the tests succeed, they are deleted by
178 default. To override this behaviour, pass `--noclean` to pytest.
179
180 The directory name starts with the system test name, followed by `-tmp-XXXXXX`,
181 i.e. `dns64-tmp-r07vei9s` for `dns64` test run. Since this name changes each
182 run, a convenience symlink that has a stable name is also created. It points to
183 the latest test artifacts directory and has a form of `dns64-sh_dns64`
184 (depending on the particular test module).
185
186 To clean up the temporary directories and symlinks, run `make clean-local` in
187 the system test directory.
188
189 The following test artifacts are typically available:
190
191 - pytest.log.txt: main log file with test output
192 - files generated by the test itself, e.g. output from "dig" and "rndc"
193 - files produced by named, other tools or helper scripts
194
195
196 ## Writing System Tests
197
198 ### File Overview
199
200 Tests are organized into system test directories which may hold one or more
201 test modules (python files). Each module may have multiple test cases. The
202 system test directories may contain the following standard files:
203
204 - `tests_*.py`: These python files are picked up by pytest as modules. If they
205 contain any test functions, they're added to the test suite.
206
207 - `*.j2`: Jinja2 templates, rendered automatically during test setup (see
208 [Templates](#templates) below).
209
210 - `ns<N>`: These subdirectories contain test name servers that can be queried
211 or can interact with each other. The value of N indicates the address the
212 server listens on: for example, ns2 listens on 10.53.0.2, and ns4 on
213 10.53.0.4. All test servers use an unprivileged port, so they don't need to
214 run as root. These servers log at the highest debug level and the log is
215 captured in the file "named.run".
216
217 - `ans<N>`: Like ns<N>, but these are mock name servers implemented in python
218 (`ans.py`), usually with the `isctest.asyncserver` module. They are
219 generally programmed to misbehave in ways named would not, so as to exercise
220 named's ability to interoperate with badly behaved name servers. A few
221 legacy mock servers are still implemented in perl (`ans.pl`); don't write
222 new ones.
223
224 The following files appear in test directories that have not yet been fully
225 ported to python; do not add them to new tests:
226
227 - `tests.sh`: Legacy shell-based tests, run via a `tests_sh_*.py` glue module.
228
229 - `setup.sh`: Legacy shell test setup. New tests use templates and a
230 `bootstrap()` function instead.
231
232 ### Module Scope
233
234 A module is a python file which contains test functions. Every system
235 test directory may contain multiple modules (i.e. tests_*.py files).
236
237 The server setup/teardown is performed for each module. Bundling test cases
238 together inside a single module may save some resources. However, test cases
239 inside a single module can't be executed in parallel.
240
241 It is possible to execute different modules defined within a single system test
242 directory in parallel. This is possible thanks to executing the tests inside a
243 temporary directory and proper port assignment to ensure there won't be any
244 conflicts.
245
246 ### Port Usage
247
248 In order for the tests to run in parallel, each test requires a unique set of
249 ports. This is ensured by the pytest runner, which assigns a unique set of
250 ports to each test module.
251
252 Inside the python tests, it is possible to use fixtures like `named_port` to
253 get the assigned port numbers. They're also set as environment variables.
254 These include:
255
256 - `PORT`: used as the basic dns port
257 - `TLSPORT`: used as the port for DNS-over-TLS
258 - `HTTPPORT`, `HTTPSPORT`: used as the ports for DNS-over-HTTP(S)
259 - `CONTROLPORT`: used as the RNDC control port
260 - `EXTRAPORT1` through `EXTRAPORT8`: additional ports that can be used as needed
261
262 ### Templates
263
264 Configuration files which need values that are only known at test run time
265 ports, default crypto algorithms, conditional sections are written as jinja2
266 templates with a `.j2` extension. During test setup, the pytest runner
267 renders every `*.j2` file in the test directory and strips the extension:
268 `ns1/named.conf.j2` becomes `ns1/named.conf`.
269
270 Inside a template, all the runner's environment variables are available with
271 `@...@` delimiters, e.g.:
272
273 ```jinja
274 options {
275 {% include_indented "_common/options.conf.j2" %}
276 tls-port @TLSPORT@;
277 };
278 ```
279
280 Standard jinja2 block syntax (`{% if %}`, `{% for %}`, ) can be used for
281 conditional or repeated sections.
282
283 Custom template variables come from an optional module-level `bootstrap()`
284 function. When a test module defines one, the runner calls it before
285 rendering the templates and passes the returned dict to the template engine:
286
287 ```python
288 def bootstrap():
289 return {"valid": True}
290 ```
291
292 Templates using custom variables must always provide defaults, so that the
293 file also renders when no value is supplied (e.g. when another module in the
294 same directory has no `bootstrap()`):
295
296 ```jinja
297 {% set valid = valid | default(False) %}
298 ```
299
300 `bootstrap()` is also the place where a module generates test data that has
301 to exist before the servers start typically zone files and DNSSEC keys (see
302 [the cookbook](COOKBOOK.md) for a complete example).
303
304 Templates can also be re-rendered while the test is running, using the
305 `templates` fixture, e.g. to change a server's config before reloading it:
306
307 ```python
308 def test_reload(ns1, templates):
309 templates.render("ns1/named.conf", {"valid": True})
310 ns1.reconfigure()
311 ```
312
313 If you don't need the file to be auto-templated during test setup, use the
314 extension `.j2.manual` instead; such templates are only rendered when the test
315 calls `templates.render()` explicitly, and no defaults are needed.
316
317 ### Fixtures and Helpers
318
319 Fixtures defined in `conftest.py` provide the test context:
320
321 - `servers` is a dictionary of all started `isctest.instance.NamedInstance`
322 servers, keyed by directory name; the shortcut fixtures `ns1` through `ns11`
323 return the corresponding instance directly. A `NamedInstance` is the
324 interface for driving a server: `ns1.rndc("...")`, `ns1.reconfigure()`,
325 `ns1.nsupdate(...)`, `ns1.watch_log_from_here()`, `ns1.ip`, ...
326 - `templates` renders jinja2 templates at runtime (see above).
327 - `named_port`, `named_tlsport`, `control_port`, ... return the assigned port
328 numbers.
329 - `system_test_dir` is the temporary directory the module runs in.
330
331 The `isctest` package provides the helper library; the modules most tests
332 need are `isctest.query` (send DNS queries), `isctest.check` (assert on
333 responses), `isctest.zone` (zone and key setup), `isctest.kasp` (DNSSEC
334 key state checks), `isctest.asyncserver` (mock servers) and `isctest.log`
335 (logging and log watchers).
336
337 Pytest marks control test collection and setup:
338
339 - `@pytest.mark.extra_artifacts([...])` declares the files (globs) the test is
340 expected to leave behind in addition to the common ones; undeclared
341 leftovers fail the run.
342 - `@pytest.mark.requires_zones_loaded("ns1", ...)` delays the test until the
343 listed servers have loaded all zones.
344 - `isctest.mark` has skip-unless conditions for environment prerequisites,
345 e.g. `isctest.mark.with_dnstap`, `isctest.mark.with_lmdb`,
346 `isctest.mark.live_internet_test`.
347
348 ### Parametrization
349
350 Use `pytest.mark.parametrize` to run one test function over several inputs
351 instead of copy-pasting test cases or looping inside one test function each
352 case is reported (and can be re-run) individually:
353
354 ```python
355 @pytest.mark.parametrize(
356 "qname,rdtype",
357 [
358 ("exists.example.", "A"),
359 ("exists.example.", "TXT"),
360 ("other.example.", "A"),
361 ],
362 )
363 def test_answers(qname, rdtype, ns1):
364 msg = isctest.query.create(qname, rdtype)
365 response = isctest.query.udp(msg, ns1.ip)
366 isctest.check.noerror(response)
367 ```
368
369 This creates the test cases `test_answers[exists.example.-A]` etc., which can
370 be passed to pytest as node IDs.
371
372 ### Logging
373
374 Each module has a separate log which will be saved as pytest.log.txt in the
375 temporary directory in which the test is executed. This log includes messages
376 for this module setup/teardown as well as any logging from the tests. Logging
377 level DEBUG and above will be present in this log.
378
379 Use `isctest.log` for test output (`isctest.log.info("...")` etc.); in
380 general, any log messages using INFO or above will also be printed out during
381 pytest execution. In CI, the pytest output is also saved to pytest.out.txt in
382 the bin/tests/system directory.
383
384 ### Adding a Test to the System Test Suite
385
386 Once a test has been created it will be automatically picked up by the pytest
387 runner if it upholds the convention expected by pytest (especially when it
388 comes to naming files and test functions). A new system test directory also
389 needs to be added to `TESTS` in `Makefile.am`, in order to be included in
390 `make check` runs.
391
392
393 ### Writing a regression reproducer
394
395 The goal: turn "issue #NNNN" into a failing test with minimal ceremony.
396
397 1. **Decide the server topology.** Most reproducers need one of:
398 - a single authoritative `named` (answer content bugs) the skeleton
399 recipe above;
400 - a resolver plus a mock server that misbehaves (resolver bugs) see
401 the mock server recipe below;
402 - signed zones and a validating resolver (DNSSEC bugs) see the zone
403 setup recipe below.
404
405 2. **Find the closest existing test and copy its shape.** Good exemplars:
406 `dispatch` (resolver + python mock server), `dnssec_py` (signed zones,
407 validator, multiple modules sharing one server set), `nsec3` (multi-module
408 family), `kasp`/`rollover_*` (key management state machines).
409
410 3. **Decide where the test lives.** If an existing directory already has the
411 server set you need, add a new `tests_*.py` module there; otherwise create
412 a new directory. Each module gets its own temporary directory, port
413 range, and parallel slot, so you are not entangled with the other modules.
414 Test functions *within* a module, however, run in file order against the
415 same live servers: a new test inherits whatever state the tests above it
416 left behind (cache contents, dynamic updates) and can disturb the tests
417 below it.
418
419 4. **Write the test to fail first.** Run it against an unfixed build and
420 make sure it fails for the reason the issue describes `ns*/named.run`
421 in the kept temporary directory is the place to verify that. Then apply
422 the fix and watch it pass.
423
424
425 ### Mock a misbehaving server
426
427 When a test needs a server that answers in ways named never would (bogus
428 glue, truncation, dropped queries, malformed records), add an `ansN`
429 subdirectory containing an `ans.py` script based on `isctest.asyncserver`.
430 The runner starts it automatically on 10.53.0.N, logging to `ans.run`.
431
432 Implementing a custom `ansN` server happens in two phases:
433
434 - define all static DNS data that the server needs to serve (if any) in `*.db`
435 files, like you would for a regular `named` instance,
436
437 - implement any non-standard behavior (modifying zone-based responses or
438 generating responses from scratch) by defining a response handler class,
439 scoping it to the QNAMEs/QTYPEs/domains it owns, and installing it into an
440 `AsyncDnsServer`.
441
442 Most importantly, avoid the temptation to define all DNS responses that a given
443 `ansN` server needs to serve using just dnspython APIs; zone files are much
444 easier to follow for static DNS data. Splitting up static DNS data and custom
445 behavior also makes it easier to follow the idea behind each test.
446
447 The most commonly subclassed handler classes are (ordered by descending
448 specificity):
449
450 - `QnameQtypeHandler`
451 - `QnameHandler`
452 - `DomainHandler`
453
454 These handler classes require certain properties (e.g. `qnames`, `qtypes`,
455 `domains`) to be defined by their subclasses. These properties define the set
456 of queries that a given handler should be used for. Please see
457 `isctest/asyncserver.py` for up-to-date information on available handler classes
458 and existing `ans.py` files for how they can be used in practice. Consult the
459 log files (`ans.run`) in case a query is not matched by its intended handler.
460
461 **NOTE:** For readability (of both code and logs), defining separate handler
462 classes for distinct queries is strongly preferred over using a single handler
463 containing an `if`/`elif`/`else` chain.
464
465 **NOTE:** If you find yourself implementing an `__init__()` method in your
466 handler subclass, it often indicates that you're approaching the problem at hand
467 from the wrong side; contact QA for guidance in such a case.
468
469 When a query is matched to a handler, the latter is expected to yield a response
470 action through its `get_responses()` method, an async generator that inspects
471 the query context and decides how the server should react:
472
473 ```python
474 from collections.abc import AsyncGenerator
475
476 import dns.flags
477
478 from isctest.asyncserver import (
479 AsyncDnsServer,
480 DnsResponseSend,
481 DomainHandler,
482 QueryContext,
483 ResponseAction,
484 )
485
486
487 class TruncateHandler(DomainHandler):
488 """Answer everything under broken.example. with TC=1."""
489
490 domains = ["broken.example."]
491
492 async def get_responses(
493 self, qctx: QueryContext
494 ) -> AsyncGenerator[ResponseAction, None]:
495 qctx.response.flags |= dns.flags.TC
496 yield DnsResponseSend(qctx.response)
497
498
499 def main() -> None:
500 server = AsyncDnsServer()
501 server.install_response_handler(TruncateHandler())
502 server.run()
503
504
505 if __name__ == "__main__":
506 main()
507 ```
508
509 The available response actions are `DnsResponseSend` (optionally with a
510 `delay`), `ResponseDrop` (don't answer at all), `BytesResponseSend` (raw
511 bytes, for malformed packets) and `CloseConnection` (TCP). Queries that no
512 handler matches are answered from zone data `AsyncDnsServer` loads every
513 `*.db` zone file found in the `ansN` directory at startup or with the
514 server's default rcode (REFUSED unless configured otherwise).
515
516 **NOTE:** For returning static responses, subclassing `StaticResponseHandler` is
517 strongly recommended instead of implementing the `get_responses()` generator
518 manually; see `resolver/ans3/ans.py` for practical examples.
519
520 **NOTE:** Calling `yield` does **NOT** make `get_responses()` return! This is
521 by design: `get_responses()` can yield multiple DNS messages in response to a
522 single query, so that it can also handle AXFR/IXFR queries, among others. Be
523 careful not to unintentionally cause multiple DNS messages to be returned for a
524 single query. If your handler's `get_responses()` method contains multiple
525 `yield` statements, it might be a sign that it needs to be refactored into
526 multiple separate handlers.
527
528 If multiple `ansN` instances used in a given system test need to share common
529 logic, extract that logic into a `<test-name>_ans.py` module in the system test
530 directory. See the `qmin` system test for a practical example.
531
532 If multiple system tests would benefit from sharing some common logic, consider
533 submitting a merge request adding that logic to `isctest/asyncserver.py` itself.
534
535 To the extent possible, try to keep each `ans.py` file limited in length and
536 scope. Look at existing `ans.py` files to see what is meant by that. If the
537 response generation logic required for reproducing a given bug is particularly
538 complex, consider dedicating the entire `ans.py` file just to that logic instead
539 of appending it to an existing one; `ansN` instances are cheap to spawn and run
540 compared to regular `named` instances. If the number of `ansN` instances used
541 in a given system test is becoming unwieldy, it usually indicates the need to
542 start adding/moving code to a new system test directory.
543
544 In some rare cases, it may be useful to reuse a common set of `nsN` server
545 instances to reproduce a whole class of related issues, triggering which relies
546 on some non-standard behavior and therefore needs a custom `ansN` server to be
547 implemented. If the logic necessary for reproducing each of these issues is
548 complex and the amount of those issues makes it impractical to add a separate
549 `ansN` server for each issue (as recommended in the previous paragraph), it is
550 acceptable to split up the test logic for each issue into separate `ans_*.py`
551 modules inside a single `ansN` directory and reduce `ans.py` itself to a loader
552 that imports and installs handlers defined in those separate modules:
553
554 ```python
555 from mytest.ans1 import ans_some_bug, ans_some_other_bug
556 from isctest.asyncserver import AsyncDnsServer
557
558
559 def main() -> None:
560 server = AsyncDnsServer()
561 server.install_response_handler(ans_some_bug.SomeBugHandler())
562 server.install_response_handler(ans_some_other_bug.SomeOtherBugHandler())
563 server.run()
564
565
566 if __name__ == "__main__":
567 main()
568 ```
569
570 However, in such a case it is particularly important to ensure consistency
571 between the names of all the Python files related to a given issue - otherwise,
572 chaos ensues. Furthermore, avoid using cryptic file names (e.g. numeric bug
573 identifiers). The recommended naming scheme is:
574
575 ```
576 mytest/
577 ans1
578 ans.py
579 ans_some_bug.py
580 ans_some_other_bug.py
581 ns2
582 ...
583 tests_some_bug.py
584 tests_some_other_bug.py
585 ```
586
587 To point a resolver at the mock, delegate to it from the test's root zone
588 (served by ns1) or list it as a forwarder; `dispatch` shows the
589 delegation pattern end to end.
590
591 The existing mock servers are the best reference. To find them, grep for
592 what you're about to use:
593 `git grep -l isctest.asyncserver -- '*/ans*/ans.py'` lists every python
594 mock, and a grep for the base class
595 (`DomainHandler`, `QnameHandler`, `ConnectionHandler`) or the response
596 action (`ResponseDrop`, `BytesResponseSend`, ...) you need usually turns
597 up a test already doing something similar. The full toolbox lives in
598 `isctest/asyncserver.py` (query matching, TCP connection handling, TSIG
599 keyrings).
600
601
602 ## Nameservers
603
604 As noted earlier, a system test will involve a number of nameservers. These
605 will be either instances of named, or mock servers, typically written in
606 Python.
607
608 For the former, the version of "named" being run is the one from the build
609 tree set up by `./configure` (i.e. if the tests are run immediately after
610 `make`, the version of "named" used is the one just built). The
611 configuration files, zone files etc. for these servers are located in
612 subdirectories of the test directory named "nsN", where N is a small integer.
613 The latter are special nameservers, mostly used for generating deliberately bad
614 responses, located in subdirectories named "ansN" (again, N is an integer).
615 In addition to configuration files, these directories should hold the
616 appropriate script files as well.
617
618 Note that the "N" for a particular test forms a single number space, e.g. if
619 there is an "ns2" directory, there cannot be an "ans2" directory as well.
620 Ideally, the directory numbers should start at 1 and work upwards.
621
622 When tests are executed, pytest takes care of the test setup and teardown. It
623 looks for any `nsN` and `ansN` directories in the system test directory and
624 starts those servers.
625
626 ### `named` Command-Line Options
627
628 By default, `named` server is started with the following options:
629
630 ```
631 -c named.conf Specifies the configuration file to use (so by implication,
632 each "nsN" nameserver's configuration file must be called
633 named.conf).
634
635 -d 99 Sets the maximum debugging level.
636
637 -D <name> The "-D" option sets a string used to identify the
638 nameserver in a process listing. In this case, the string
639 is the name of the subdirectory.
640
641 -g Runs the server in the foreground and logs everything to
642 stderr.
643
644 -m record
645 Turns on these memory usage debugging flags.
646 ```
647
648 All output is sent to a file called `named.run` in the nameserver directory.
649
650 The options used to start named can be altered. There are a couple ways of
651 doing this. The runner checks the methods in a specific order: if a check
652 succeeds, the options are set and any other specification is ignored. In
653 order, these are:
654
655 1. Including a file called "named.args" in the "nsN" directory. If present,
656 the contents of the first non-commented, non-blank line of the file are used as
657 the named command-line arguments. The rest of the file is ignored.
658
659 2. Tweaking the default command line arguments with "-T" options. This flag is
660 used to alter the behavior of BIND for testing and is not documented in the
661 ARM. The presence of a file called `named.<flag>` in the "nsN" directory adds
662 `-T <flag>` to the default command line (the content of the file is irrelevant
663 - it is only the presence that counts). The recognized flags are:
664
665 ```
666 dropedns Recognise EDNS options in messages, but drop messages
667 containing them.
668
669 ednsformerr, ednsnotimp, ednsrefused
670 Answer EDNS queries with the given rcode, pretending to
671 be an old server that doesn't understand EDNS.
672
673 cookiealwaysvalid
674 Accept any DNS cookie presented by a client.
675
676 noaa Never set the AA bit in an answer.
677
678 noedns Disable recognition of EDNS options in messages.
679
680 nonearest Omit the closest-encloser NSEC3 proof from negative
681 responses (except for DS queries).
682
683 nosoa Disable the addition of SOA records to negative
684 responses (or to the additional section if the response
685 is triggered by RPZ rewriting).
686
687 maxudp512, maxudp1460
688 Set the maximum UDP size handled by named to 512/1460.
689
690 tat=1, tat=3 Send trust-anchor-telemetry queries every N seconds.
691
692 notcp Disable TCP in "named". Unlike the other flags, this
693 one is also applied when "named.args" is used.
694 ```
695
696 ### Running Nameservers Interactively
697
698 In order to debug the nameservers, you can let pytest perform the nameserver
699 setup and interact with the servers before the test starts, or even at specific
700 points during the test, using the `--trace` option to drop you into pdb debugger
701 which pauses the execution of the tests, while keeping the server state intact:
702
703 ```sh
704 pytest -k dns64 --trace
705 ```
706
707
708 ## Developer Notes
709
710 ### Test discovery and collection
711
712 There are two distinct types of system tests. The first is a legacy shell
713 script tests.sh containing individual test cases executed sequentially and the
714 success/failure is determined by return code. The second type is a regular
715 pytest file which contains test functions.
716
717 Dealing with the regular pytest files doesn't require any special consideration
718 as long as the naming conventions are met. Discovering the tests.sh tests is
719 more complicated.
720
721 The chosen solution is to add a bit of glue for each system test. For every
722 tests.sh, there is an accompanying tests_sh_*.py file that contains a test
723 function which utilizes a custom run_tests_sh fixture to call the tests.sh
724 script. Other solutions were tried and eventually rejected. While this
725 introduces a bit of extra glue, it is the most portable, compatible and least
726 complex solution.
727
728 ### Compatibility with older pytest version
729
730 The minimum supported versions of python and the required python packages are
731 declared in [requirements.txt](requirements.txt) and in the
732 `pytest_configure()` check in `conftest.py`. When implementing new runner
733 features, check feature support in the pytest and pytest-xdist versions
734 available in the oldest distributions covered by CI first; we may need to add
735 compat code to handle breaking upstream changes in either direction.
736
737 ### Format of Shell Test Output
738
739 Legacy shell-based tests have the following format of output:
740
741 ```
742 <letter>:<test-name>:<message> [(<number>)]
743 ```
744
745 e.g.
746
747 ```
748 I:catz:checking that dom1.example is not served by primary (1)
749 ```
750
751 The meanings of the fields are as follows:
752
753 <letter>
754 This indicates the type of message. This is one of:
755
756 ```
757 S Start of the test
758 A Start of test (retained for backwards compatibility)
759 T Start of test (retained for backwards compatibility)
760 E End of the test
761 I Information. A test will typically output many of these messages
762 during its run, indicating test progress. Note that such a message may
763 be of the form "I:testname:failed", indicating that a sub-test has
764 failed.
765 R Result. Each test will result in one such message, which is of the
766 form:
767
768 R:<test-tmpdir>:<result>
769
770 where <result> is one of:
771
772 PASS The test passed
773 FAIL The test failed
774 SKIPPED The test was not run, usually because some
775 prerequisites required to run the test are missing.
776 ```
777
778 <test-tmpdir>
779 This is the name of the temporary test directory from which the message
780 emanated, which is also the name of the subdirectory holding the test files.
781
782 <message>
783 This is text output by the test during its execution.
784
785 (<number>)
786 If present, this will correlate with a file created by the test. The tests
787 execute commands and route the output of each command to a file. The name of
788 this file depends on the command and the test, but will usually be of the form:
789
790 ```
791 <command>.out.<suffix><number>
792 ```
793
794 e.g. nsupdate.out.test28, dig.out.q3. This aids diagnosis of problems by
795 allowing the output that caused the problem message to be identified.
796