Home | History | Annotate | Line # | Download | only in system
      1 <!--
      2 Copyright (C) Internet Systems Consortium, Inc. ("ISC")
      3 
      4 SPDX-License-Identifier: MPL-2.0
      5 
      6 This Source Code Form is subject to the terms of the Mozilla Public
      7 License, v. 2.0.  If a copy of the MPL was not distributed with this
      8 file, you can obtain one at https://mozilla.org/MPL/2.0/.
      9 
     10 See the COPYRIGHT file distributed with this work for additional
     11 information regarding copyright ownership.
     12 -->
     13 
     14 # BIND9 System Test Cookbook
     15 
     16 Task-oriented recipes for the system test framework.  Each recipe is
     17 self-contained and shows working code that can be copied and adapted.
     18 For concepts and reference documentation (build setup, fixtures, runner
     19 internals), see [README.md](README.md).
     20 
     21 Prerequisite for all recipes: a built tree and the test network interfaces
     22 configured  see "Running the Tests" in the README.
     23 
     24 
     25 ## Iterate on a single test
     26 
     27 The basic loop while developing a test:
     28 
     29 ```sh
     30 cd bin/tests/system
     31 pytest mytest                                     # whole directory
     32 pytest mytest/tests_mytestmod.py::test_one_case   # one test function
     33 ```
     34 
     35 When a test fails, its temporary directory is kept and a stable symlink to it
     36 is created (e.g. `mytest-mytestmod` for `mytest/tests_mytestmod.py`).  Look
     37 there for `pytest.log.txt` (the test's own log) and `ns*/named.run` (each
     38 server's debug-level log).  Pass `--noclean` to keep the directory even on
     39 success.
     40 
     41 To poke at the servers interactively, pause the test at any point by
     42 inserting a breakpoint, or start pdb right away with `--trace`; to get a pdb
     43 prompt automatically when a test fails, use `--pdb`.  In all cases the
     44 servers keep running while the test is paused, so their state can be
     45 inspected:
     46 
     47 ```sh
     48 pytest mytest --trace   # pdb at the start of each test
     49 pytest mytest --pdb     # pdb when a test fails
     50 ```
     51 
     52 Timing-sensitive bugs hide in system tests: before declaring a test done, run
     53 it several times (and ideally once under parallel load, `pytest -n auto`).
     54 A test that flakes is treated as a bug in the test.
     55 
     56 
     57 ## Add a new system test directory
     58 
     59 This skeleton sets up one authoritative server and queries it.  Pick a name
     60 that starts with a letter and uses underscores as word separators (hyphens
     61 are not allowed), and create:
     62 
     63 ```
     64 demo/
     65  ns1/
     66     named.conf.j2
     67     example.db
     68  tests_demo.py
     69 ```
     70 
     71 `demo/ns1/named.conf.j2`  the config template; the runner renders it to
     72 `named.conf` at setup time, filling in the assigned ports:
     73 
     74 ```jinja
     75 options {
     76 	{% include_indented "_common/options.conf.j2" %}
     77 	recursion no;
     78 	dnssec-validation no;
     79 };
     80 
     81 {% include "_common/controls.conf.j2" %}
     82 
     83 zone "example" {
     84 	type primary;
     85 	file "example.db";
     86 };
     87 ```
     88 
     89 The `_common/options.conf.j2` include expands to the standard plumbing
     90 (configuring interfaces, source addresses, port and pid file).  Write only the
     91 options your test actually cares about below the include.  When the standard
     92 block doesn't fit, compose the pieces from `_common/options/` instead and put
     93 anything nonstandard inline.
     94 
     95 The plumbing templates are built on the `ns` variable: templates inside an
     96 `nsN`/`ansN` subdirectory automatically get one describing that server, so a
     97 config never hardcodes its own address  `@ns.ip@` renders to 10.53.0.1 in
     98 ns1, 10.53.0.2 in ns2, ..., and `@ns.ip6@` to the matching
     99 fd92:7065:b8e:ffff::N address.  Use it wherever a config refers to the
    100 instance's own address; references to *other* servers stay literal.
    101 
    102 The `_common/controls.conf.j2` include sets up the rndc control channel, so
    103 the test (and the runner's shutdown sequence) can use `rndc`.  A resolver
    104 instance would also include `_common/root.hint.conf` to get the standard
    105 root hints (ns1 is the root server by convention).  Inside an indented
    106 section such as a view statement, use `{% include_indented "..." %}` instead
    107 of `{% include %}`  it aligns the inserted block with the tag's own
    108 indentation.
    109 
    110 `demo/ns1/example.db`  a plain zone file:
    111 
    112 ```
    113 $TTL 300
    114 example.        IN SOA  ns1.example. hostmaster.example. 1 600 600 1200 600
    115 example.        NS      ns1.example.
    116 ns1.example.    A       10.53.0.1
    117 a.example.      A       10.0.0.1
    118 ```
    119 
    120 `demo/tests_demo.py`  the test module:
    121 
    122 ```python
    123 import isctest
    124 
    125 
    126 def test_a_record(ns1):
    127     msg = isctest.query.create("a.example.", "A")
    128     response = isctest.query.udp(msg, ns1.ip)
    129     isctest.check.noerror(response)
    130     assert str(response.answer[0][0]) == "10.0.0.1"
    131 ```
    132 
    133 That's it  run it with `pytest demo`.  The pytest runner discovers new
    134 directories automatically; to also include the test in `make check` runs,
    135 add it to `TESTS` in `Makefile.am`.
    136 
    137 Notes:
    138 
    139 - The `ns1` fixture is the started server (an
    140   `isctest.instance.NamedInstance`); `ns1.ip` is 10.53.0.1.  Servers in
    141   `nsN`/`ansN` subdirectories are started automatically.
    142 - If your test writes files beyond the usual ones (e.g. `dig.out.*`,
    143   `dsset-*`), declare them as a module-level marker 
    144   `pytestmark = pytest.mark.extra_artifacts(["dig.out.*"])`  or the
    145   artifact check at teardown will fail.  Most real test modules carry one.
    146 
    147 
    148 ## Set up zones in bootstrap()
    149 
    150 A module-level `bootstrap()` function runs before the config templates are
    151 rendered and before the servers start.  This is where zone files and DNSSEC
    152 keys can be generated; whatever dict it returns becomes template data.
    153 
    154 The simplest case is a plain unsigned zone  use this instead of writing
    155 the same boilerplate zone file by hand in every `nsN` directory:
    156 
    157 ```python
    158 from isctest.template import NS1, zones
    159 from isctest.zone import Zone
    160 
    161 
    162 def bootstrap():
    163     zone = Zone("example", NS1)    # signed=False is the default
    164     zone.configure()               # render the zone file
    165     return {"zones": zones([zone])}
    166 ```
    167 
    168 `Zone.configure()` renders the zone file from
    169 `_common/zones/template.db.j2.manual`, a generic zone with SOA, NS, and a
    170 few test records.  To provide your own content, drop a template named after
    171 the zone file  `ns1/zones/<name>.db.j2.manual`  and `configure()` picks
    172 it up automatically instead of the generic one.  (The `.j2.manual` suffix
    173 keeps the file out of the runner's automatic template pass, which renders
    174 every plain `*.j2` at setup time; here `bootstrap()` renders it instead.)
    175 
    176 For DNSSEC-signed zones, pass `signed=True` and make sure to configure the
    177 `trust_anchors`:
    178 
    179 ```python
    180 from isctest.template import NS1, NS2, zones
    181 from isctest.zone import Zone, configure_root
    182 
    183 
    184 def bootstrap():
    185     zone = Zone("signed-example", NS2, signed=True)
    186     zone.configure()           # keys, zone file, dnssec-signzone
    187 
    188     # root zone on ns1, delegating to (and providing DS records for)
    189     # the zone above
    190     root = configure_root([zone])
    191 
    192     return {
    193         "trust_anchors": root.trust_anchors(),
    194         "zones": zones([root, zone]),
    195     }
    196 ```
    197 
    198 With `signed=True`, `configure()` also generates a KSK+ZSK with
    199 `dnssec-keygen` and signs the rendered zone with `dnssec-signzone`.  To
    200 sign a hand-written zone file checked into git (no rendering at all), put
    201 it at `ns2/zones/<name>.db` and drive the steps yourself:
    202 
    203 ```python
    204 def bootstrap():
    205     zone = Zone("signed-example", NS2, signed=True)
    206     zone.add_keys()
    207     zone.sign("-3 ABCD")       # extra dnssec-signzone args, e.g. NSEC3
    208 
    209     root = configure_root([zone])
    210     return {
    211         "trust_anchors": root.trust_anchors(),
    212         "zones": zones([root, zone]),
    213     }
    214 ```
    215 
    216 On the template side, the `_common` includes consume the returned data: the
    217 authoritative server's `named.conf.j2` ends with
    218 
    219 ```jinja
    220 {% include "_common/controls.conf.j2" %}
    221 {% include "_common/zones.conf.j2" %}
    222 ```
    223 
    224 (`zones.conf.j2` emits a `zone` block for every zone whose nameserver
    225 matches the rendering server) and the validating resolver's with
    226 
    227 ```jinja
    228 {% include "_common/controls.conf.j2" %}
    229 {% include "_common/trusted.conf.j2" %}
    230 {% include "_common/root.hint.conf" %}
    231 ```
    232 
    233 (`trusted.conf.j2` emits `trust-anchors` from the `trust_anchors` data, and
    234 `root.hint.conf` points the resolver at ns1 for the root zone).
    235 
    236 The `dnssec_py` directory is the canonical example of this pattern,
    237 including several modules with different zone setups sharing one directory.
    238 Declare the generated files as artifacts:
    239 
    240 ```python
    241 pytestmark = pytest.mark.extra_artifacts(
    242     [
    243         "ns*/dsset-*",
    244         "ns*/trusted.conf",
    245         "ns*/zones/*.db",
    246         "ns*/zones/*.db.signed",
    247     ]
    248 )
    249 ```
    250 
    251 
    252 ## Drive named and watch its logs
    253 
    254 The `NamedInstance` fixtures (`ns1` ... `ns11`) control the running servers.
    255 The cardinal rule: never `time.sleep()` waiting for the server  watch the
    256 log for the line that proves the event happened.
    257 
    258 Wait for a log line caused by an action  enter the watcher *before*
    259 triggering the action, so the line cannot slip past unobserved:
    260 
    261 ```python
    262 def test_cache_flush(ns4):
    263     with ns4.watch_log_from_here() as watcher:
    264         ns4.rndc("flush")
    265         watcher.wait_for_line("flushing caches in all views succeeded")
    266 ```
    267 
    268 `wait_for_line()` accepts a string, a compiled regex, or a list of either
    269 (any match wins) and returns the `re.Match`.  `wait_for_all([...])` waits
    270 until every pattern has appeared, `wait_for_sequence([...])` requires them
    271 in order.  To match lines that may already have been logged (e.g. during
    272 startup), use `watch_log_from_start()` instead.
    273 
    274 Reconfigure a server mid-test by re-rendering its config template with new
    275 data, then reloading.  `reconfigure()` and `reload()` send the rndc command
    276 and wait for the corresponding completion line in the log:
    277 
    278 ```python
    279 def test_with_new_config(ns1, templates):
    280     templates.render("ns1/named.conf", {"flag": True})
    281     ns1.reconfigure()
    282 ```
    283 
    284 Dynamic updates go through `nsupdate`:
    285 
    286 ```python
    287 import dns.update
    288 
    289 def test_update(ns1):
    290     update = dns.update.UpdateMessage("example.")
    291     update.add("added.example.", 300, "A", "10.0.0.2")
    292     ns1.nsupdate(update)
    293 ```
    294 
    295 Send queries and check the responses with `isctest.query` and
    296 `isctest.check`:
    297 
    298 ```python
    299 msg = isctest.query.create("a.example.", "A")
    300 response = isctest.query.udp(msg, ns1.ip)     # or .tcp(), .tls()
    301 isctest.check.noerror(response)               # rcode checks: nxdomain,
    302                                               #  servfail, refused, ...
    303 isctest.check.adflag(response)                # flag checks: rdflag,
    304                                               #  noraflag, ...
    305 isctest.check.same_answer(response, expected) # compare two responses
    306 ```
    307 
    308 To wait until a zone transfer has happened, poll the SOA serial:
    309 
    310 ```python
    311 isctest.query.wait_for_serial(ns2.ip, "example.", expected_serial)
    312 ```
    313 
    314 For grepping a file the server produced (dumps, statistics), use
    315 `isctest.text`:
    316 
    317 ```python
    318 from re import compile as Re
    319 
    320 ns4.rndc("dumpdb -all")
    321 dump = isctest.text.TextFile("ns4/named_dump.db")
    322 assert dump.grep(Re(r"^a\.example\..*10\.0\.0\.1"))
    323 ```
    324