Home | History | Annotate | Line # | Download | only in system
COOKBOOK.md revision 1.1
      1 <!--
      2 Copyright (C) Internet Systems Consortium, Inc. ("ISC")
      3 
      4 SPDX-License-Identifier: MPL-2.0
      5 
      6 This Source Code Form is subject to the terms of the Mozilla Public
      7 License, v. 2.0.  If a copy of the MPL was not distributed with this
      8 file, you can obtain one at https://mozilla.org/MPL/2.0/.
      9 
     10 See the COPYRIGHT file distributed with this work for additional
     11 information regarding copyright ownership.
     12 -->
     13 
     14 # BIND9 System Test Cookbook
     15 
     16 Task-oriented recipes for the system test framework.  Each recipe is
     17 self-contained and shows working code that can be copied and adapted.
     18 For concepts and reference documentation (build setup, fixtures, runner
     19 internals), see [README.md](README.md).
     20 
     21 Prerequisite for all recipes: a built tree and the test network interfaces
     22 configured  see "Running the Tests" in the README.
     23 
     24 
     25 ## Iterate on a single test
     26 
     27 The basic loop while developing a test:
     28 
     29 ```sh
     30 cd bin/tests/system
     31 pytest mytest                                     # whole directory
     32 pytest mytest/tests_mytestmod.py::test_one_case   # one test function
     33 ```
     34 
     35 When a test fails, its temporary directory is kept and a stable symlink to it
     36 is created (e.g. `mytest-mytestmod` for `mytest/tests_mytestmod.py`).  Look
     37 there for `pytest.log.txt` (the test's own log) and `ns*/named.run` (each
     38 server's debug-level log).  Pass `--noclean` to keep the directory even on
     39 success.
     40 
     41 To poke at the servers interactively, pause the test at any point by
     42 inserting a breakpoint, or start pdb right away with `--trace`; to get a pdb
     43 prompt automatically when a test fails, use `--pdb`.  In all cases the
     44 servers keep running while the test is paused, so their state can be
     45 inspected:
     46 
     47 ```sh
     48 pytest mytest --trace   # pdb at the start of each test
     49 pytest mytest --pdb     # pdb when a test fails
     50 ```
     51 
     52 Timing-sensitive bugs hide in system tests: before declaring a test done, run
     53 it several times (and ideally once under parallel load, `pytest -n auto`).
     54 A test that flakes is treated as a bug in the test.
     55 
     56 
     57 ## Add a new system test directory
     58 
     59 This skeleton sets up one authoritative server and queries it.  Pick a name
     60 that starts with a letter and uses underscores as word separators (hyphens
     61 are not allowed), and create:
     62 
     63 ```
     64 demo/
     65  ns1/
     66     named.conf.j2
     67     example.db
     68  tests_demo.py
     69 ```
     70 
     71 `demo/ns1/named.conf.j2`  the config template; the runner renders it to
     72 `named.conf` at setup time, filling in the assigned ports.  Templates inside
     73 an `nsN`/`ansN` subdirectory also get an `ns` variable describing that
     74 server, so the config doesn't hardcode its own address (`@ns.ip@` renders to
     75 10.53.0.1 in ns1, 10.53.0.2 in ns2, ...):
     76 
     77 ```jinja
     78 options {
     79     query-source address @ns.ip@;
     80     notify-source @ns.ip@;
     81     transfer-source @ns.ip@;
     82     port @PORT@;
     83     pid-file "named.pid";
     84     listen-on { @ns.ip@; };
     85     listen-on-v6 { none; };
     86     recursion no;
     87     dnssec-validation no;
     88 };
     89 
     90 {% include "_common/controls.conf.j2" %}
     91 
     92 zone "example" {
     93     type primary;
     94     file "example.db";
     95 };
     96 ```
     97 
     98 The `_common/controls.conf.j2` include sets up the rndc control channel, so
     99 the test (and the runner's shutdown sequence) can use `rndc`.
    100 
    101 `demo/ns1/example.db`  a plain zone file:
    102 
    103 ```
    104 $TTL 300
    105 example.        IN SOA  ns1.example. hostmaster.example. 1 600 600 1200 600
    106 example.        NS      ns1.example.
    107 ns1.example.    A       10.53.0.1
    108 a.example.      A       10.0.0.1
    109 ```
    110 
    111 `demo/tests_demo.py`  the test module:
    112 
    113 ```python
    114 import isctest
    115 
    116 
    117 def test_a_record(ns1):
    118     msg = isctest.query.create("a.example.", "A")
    119     response = isctest.query.udp(msg, ns1.ip)
    120     isctest.check.noerror(response)
    121     assert str(response.answer[0][0]) == "10.0.0.1"
    122 ```
    123 
    124 That's it  run it with `pytest demo`.  The pytest runner discovers new
    125 directories automatically; to also include the test in `make check` runs,
    126 add it to `TESTS` in `Makefile.am`.
    127 
    128 Notes:
    129 
    130 - The `ns1` fixture is the started server (an
    131   `isctest.instance.NamedInstance`); `ns1.ip` is 10.53.0.1.  Servers in
    132   `nsN`/`ansN` subdirectories are started automatically.
    133 - If your test writes files beyond the usual ones (e.g. `dig.out.*`,
    134   `dsset-*`), declare them as a module-level marker 
    135   `pytestmark = pytest.mark.extra_artifacts(["dig.out.*"])`  or the
    136   artifact check at teardown will fail.  Most real test modules carry one.
    137 
    138 
    139 ## Set up zones in bootstrap()
    140 
    141 A module-level `bootstrap()` function runs before the config templates are
    142 rendered and before the servers start.  This is where zone files and DNSSEC
    143 keys can be generated; whatever dict it returns becomes template data.
    144 
    145 The simplest case is a plain unsigned zone  use this instead of writing
    146 the same boilerplate zone file by hand in every `nsN` directory:
    147 
    148 ```python
    149 from isctest.template import NS1, zones
    150 from isctest.zone import Zone
    151 
    152 
    153 def bootstrap():
    154     zone = Zone("example", NS1)    # signed=False is the default
    155     zone.configure()               # render the zone file
    156     return {"zones": zones([zone])}
    157 ```
    158 
    159 `Zone.configure()` renders the zone file from
    160 `_common/zones/template.db.j2.manual`, a generic zone with SOA, NS, and a
    161 few test records.  To provide your own content, drop a template named after
    162 the zone file  `ns1/zones/<name>.db.j2.manual`  and `configure()` picks
    163 it up automatically instead of the generic one.  (The `.j2.manual` suffix
    164 keeps the file out of the runner's automatic template pass, which renders
    165 every plain `*.j2` at setup time; here `bootstrap()` renders it instead.)
    166 
    167 For DNSSEC-signed zones, pass `signed=True` and make sure to configure the
    168 `trust_anchors`:
    169 
    170 ```python
    171 from isctest.template import NS1, NS2, zones
    172 from isctest.zone import Zone, configure_root
    173 
    174 
    175 def bootstrap():
    176     zone = Zone("signed-example", NS2, signed=True)
    177     zone.configure()           # keys, zone file, dnssec-signzone
    178 
    179     # root zone on ns1, delegating to (and providing DS records for)
    180     # the zone above
    181     root = configure_root([zone])
    182 
    183     return {
    184         "trust_anchors": root.trust_anchors(),
    185         "zones": zones([root, zone]),
    186     }
    187 ```
    188 
    189 With `signed=True`, `configure()` also generates a KSK+ZSK with
    190 `dnssec-keygen` and signs the rendered zone with `dnssec-signzone`.  To
    191 sign a hand-written zone file checked into git (no rendering at all), put
    192 it at `ns2/zones/<name>.db` and drive the steps yourself:
    193 
    194 ```python
    195 def bootstrap():
    196     zone = Zone("signed-example", NS2, signed=True)
    197     zone.add_keys()
    198     zone.sign("-3 ABCD")       # extra dnssec-signzone args, e.g. NSEC3
    199 
    200     root = configure_root([zone])
    201     return {
    202         "trust_anchors": root.trust_anchors(),
    203         "zones": zones([root, zone]),
    204     }
    205 ```
    206 
    207 On the template side, the `_common` includes consume the returned data: the
    208 authoritative server's `named.conf.j2` ends with
    209 
    210 ```jinja
    211 {% include "_common/controls.conf.j2" %}
    212 {% include "_common/zones.conf.j2" %}
    213 ```
    214 
    215 (`zones.conf.j2` emits a `zone` block for every zone whose nameserver
    216 matches the rendering server) and the validating resolver's with
    217 
    218 ```jinja
    219 {% include "_common/controls.conf.j2" %}
    220 {% include "_common/trusted.conf.j2" %}
    221 {% include "_common/root.hint.conf" %}
    222 ```
    223 
    224 (`trusted.conf.j2` emits `trust-anchors` from the `trust_anchors` data, and
    225 `root.hint.conf` points the resolver at ns1 for the root zone).
    226 
    227 The `dnssec_py` directory is the canonical example of this pattern,
    228 including several modules with different zone setups sharing one directory.
    229 Declare the generated files as artifacts:
    230 
    231 ```python
    232 pytestmark = pytest.mark.extra_artifacts(
    233     [
    234         "ns*/dsset-*",
    235         "ns*/trusted.conf",
    236         "ns*/zones/*.db",
    237         "ns*/zones/*.db.signed",
    238     ]
    239 )
    240 ```
    241 
    242 
    243 ## Drive named and watch its logs
    244 
    245 The `NamedInstance` fixtures (`ns1` ... `ns11`) control the running servers.
    246 The cardinal rule: never `time.sleep()` waiting for the server  watch the
    247 log for the line that proves the event happened.
    248 
    249 Wait for a log line caused by an action  enter the watcher *before*
    250 triggering the action, so the line cannot slip past unobserved:
    251 
    252 ```python
    253 def test_cache_flush(ns4):
    254     with ns4.watch_log_from_here() as watcher:
    255         ns4.rndc("flush")
    256         watcher.wait_for_line("flushing caches in all views succeeded")
    257 ```
    258 
    259 `wait_for_line()` accepts a string, a compiled regex, or a list of either
    260 (any match wins) and returns the `re.Match`.  `wait_for_all([...])` waits
    261 until every pattern has appeared, `wait_for_sequence([...])` requires them
    262 in order.  To match lines that may already have been logged (e.g. during
    263 startup), use `watch_log_from_start()` instead.
    264 
    265 Reconfigure a server mid-test by re-rendering its config template with new
    266 data, then reloading.  `reconfigure()` and `reload()` send the rndc command
    267 and wait for the corresponding completion line in the log:
    268 
    269 ```python
    270 def test_with_new_config(ns1, templates):
    271     templates.render("ns1/named.conf", {"flag": True})
    272     ns1.reconfigure()
    273 ```
    274 
    275 Dynamic updates go through `nsupdate`:
    276 
    277 ```python
    278 import dns.update
    279 
    280 def test_update(ns1):
    281     update = dns.update.UpdateMessage("example.")
    282     update.add("added.example.", 300, "A", "10.0.0.2")
    283     ns1.nsupdate(update)
    284 ```
    285 
    286 Send queries and check the responses with `isctest.query` and
    287 `isctest.check`:
    288 
    289 ```python
    290 msg = isctest.query.create("a.example.", "A")
    291 response = isctest.query.udp(msg, ns1.ip)     # or .tcp(), .tls()
    292 isctest.check.noerror(response)               # rcode checks: nxdomain,
    293                                               #  servfail, refused, ...
    294 isctest.check.adflag(response)                # flag checks: rdflag,
    295                                               #  noraflag, ...
    296 isctest.check.same_answer(response, expected) # compare two responses
    297 ```
    298 
    299 To wait until a zone transfer has happened, poll the SOA serial:
    300 
    301 ```python
    302 isctest.query.wait_for_serial(ns2.ip, "example.", expected_serial)
    303 ```
    304 
    305 For grepping a file the server produced (dumps, statistics), use
    306 `isctest.text`:
    307 
    308 ```python
    309 from re import compile as Re
    310 
    311 ns4.rndc("dumpdb -all")
    312 dump = isctest.text.TextFile("ns4/named_dump.db")
    313 assert dump.grep(Re(r"^a\.example\..*10\.0\.0\.1"))
    314 ```
    315