COOKBOOK.md revision 1.1 1 <!--
2 Copyright (C) Internet Systems Consortium, Inc. ("ISC")
3
4 SPDX-License-Identifier: MPL-2.0
5
6 This Source Code Form is subject to the terms of the Mozilla Public
7 License, v. 2.0. If a copy of the MPL was not distributed with this
8 file, you can obtain one at https://mozilla.org/MPL/2.0/.
9
10 See the COPYRIGHT file distributed with this work for additional
11 information regarding copyright ownership.
12 -->
13
14 # BIND9 System Test Cookbook
15
16 Task-oriented recipes for the system test framework. Each recipe is
17 self-contained and shows working code that can be copied and adapted.
18 For concepts and reference documentation (build setup, fixtures, runner
19 internals), see [README.md](README.md).
20
21 Prerequisite for all recipes: a built tree and the test network interfaces
22 configured see "Running the Tests" in the README.
23
24
25 ## Iterate on a single test
26
27 The basic loop while developing a test:
28
29 ```sh
30 cd bin/tests/system
31 pytest mytest # whole directory
32 pytest mytest/tests_mytestmod.py::test_one_case # one test function
33 ```
34
35 When a test fails, its temporary directory is kept and a stable symlink to it
36 is created (e.g. `mytest-mytestmod` for `mytest/tests_mytestmod.py`). Look
37 there for `pytest.log.txt` (the test's own log) and `ns*/named.run` (each
38 server's debug-level log). Pass `--noclean` to keep the directory even on
39 success.
40
41 To poke at the servers interactively, pause the test at any point by
42 inserting a breakpoint, or start pdb right away with `--trace`; to get a pdb
43 prompt automatically when a test fails, use `--pdb`. In all cases the
44 servers keep running while the test is paused, so their state can be
45 inspected:
46
47 ```sh
48 pytest mytest --trace # pdb at the start of each test
49 pytest mytest --pdb # pdb when a test fails
50 ```
51
52 Timing-sensitive bugs hide in system tests: before declaring a test done, run
53 it several times (and ideally once under parallel load, `pytest -n auto`).
54 A test that flakes is treated as a bug in the test.
55
56
57 ## Add a new system test directory
58
59 This skeleton sets up one authoritative server and queries it. Pick a name
60 that starts with a letter and uses underscores as word separators (hyphens
61 are not allowed), and create:
62
63 ```
64 demo/
65 ns1/
66 named.conf.j2
67 example.db
68 tests_demo.py
69 ```
70
71 `demo/ns1/named.conf.j2` the config template; the runner renders it to
72 `named.conf` at setup time, filling in the assigned ports. Templates inside
73 an `nsN`/`ansN` subdirectory also get an `ns` variable describing that
74 server, so the config doesn't hardcode its own address (`@ns.ip@` renders to
75 10.53.0.1 in ns1, 10.53.0.2 in ns2, ...):
76
77 ```jinja
78 options {
79 query-source address @ns.ip@;
80 notify-source @ns.ip@;
81 transfer-source @ns.ip@;
82 port @PORT@;
83 pid-file "named.pid";
84 listen-on { @ns.ip@; };
85 listen-on-v6 { none; };
86 recursion no;
87 dnssec-validation no;
88 };
89
90 {% include "_common/controls.conf.j2" %}
91
92 zone "example" {
93 type primary;
94 file "example.db";
95 };
96 ```
97
98 The `_common/controls.conf.j2` include sets up the rndc control channel, so
99 the test (and the runner's shutdown sequence) can use `rndc`.
100
101 `demo/ns1/example.db` a plain zone file:
102
103 ```
104 $TTL 300
105 example. IN SOA ns1.example. hostmaster.example. 1 600 600 1200 600
106 example. NS ns1.example.
107 ns1.example. A 10.53.0.1
108 a.example. A 10.0.0.1
109 ```
110
111 `demo/tests_demo.py` the test module:
112
113 ```python
114 import isctest
115
116
117 def test_a_record(ns1):
118 msg = isctest.query.create("a.example.", "A")
119 response = isctest.query.udp(msg, ns1.ip)
120 isctest.check.noerror(response)
121 assert str(response.answer[0][0]) == "10.0.0.1"
122 ```
123
124 That's it run it with `pytest demo`. The pytest runner discovers new
125 directories automatically; to also include the test in `make check` runs,
126 add it to `TESTS` in `Makefile.am`.
127
128 Notes:
129
130 - The `ns1` fixture is the started server (an
131 `isctest.instance.NamedInstance`); `ns1.ip` is 10.53.0.1. Servers in
132 `nsN`/`ansN` subdirectories are started automatically.
133 - If your test writes files beyond the usual ones (e.g. `dig.out.*`,
134 `dsset-*`), declare them as a module-level marker
135 `pytestmark = pytest.mark.extra_artifacts(["dig.out.*"])` or the
136 artifact check at teardown will fail. Most real test modules carry one.
137
138
139 ## Set up zones in bootstrap()
140
141 A module-level `bootstrap()` function runs before the config templates are
142 rendered and before the servers start. This is where zone files and DNSSEC
143 keys can be generated; whatever dict it returns becomes template data.
144
145 The simplest case is a plain unsigned zone use this instead of writing
146 the same boilerplate zone file by hand in every `nsN` directory:
147
148 ```python
149 from isctest.template import NS1, zones
150 from isctest.zone import Zone
151
152
153 def bootstrap():
154 zone = Zone("example", NS1) # signed=False is the default
155 zone.configure() # render the zone file
156 return {"zones": zones([zone])}
157 ```
158
159 `Zone.configure()` renders the zone file from
160 `_common/zones/template.db.j2.manual`, a generic zone with SOA, NS, and a
161 few test records. To provide your own content, drop a template named after
162 the zone file `ns1/zones/<name>.db.j2.manual` and `configure()` picks
163 it up automatically instead of the generic one. (The `.j2.manual` suffix
164 keeps the file out of the runner's automatic template pass, which renders
165 every plain `*.j2` at setup time; here `bootstrap()` renders it instead.)
166
167 For DNSSEC-signed zones, pass `signed=True` and make sure to configure the
168 `trust_anchors`:
169
170 ```python
171 from isctest.template import NS1, NS2, zones
172 from isctest.zone import Zone, configure_root
173
174
175 def bootstrap():
176 zone = Zone("signed-example", NS2, signed=True)
177 zone.configure() # keys, zone file, dnssec-signzone
178
179 # root zone on ns1, delegating to (and providing DS records for)
180 # the zone above
181 root = configure_root([zone])
182
183 return {
184 "trust_anchors": root.trust_anchors(),
185 "zones": zones([root, zone]),
186 }
187 ```
188
189 With `signed=True`, `configure()` also generates a KSK+ZSK with
190 `dnssec-keygen` and signs the rendered zone with `dnssec-signzone`. To
191 sign a hand-written zone file checked into git (no rendering at all), put
192 it at `ns2/zones/<name>.db` and drive the steps yourself:
193
194 ```python
195 def bootstrap():
196 zone = Zone("signed-example", NS2, signed=True)
197 zone.add_keys()
198 zone.sign("-3 ABCD") # extra dnssec-signzone args, e.g. NSEC3
199
200 root = configure_root([zone])
201 return {
202 "trust_anchors": root.trust_anchors(),
203 "zones": zones([root, zone]),
204 }
205 ```
206
207 On the template side, the `_common` includes consume the returned data: the
208 authoritative server's `named.conf.j2` ends with
209
210 ```jinja
211 {% include "_common/controls.conf.j2" %}
212 {% include "_common/zones.conf.j2" %}
213 ```
214
215 (`zones.conf.j2` emits a `zone` block for every zone whose nameserver
216 matches the rendering server) and the validating resolver's with
217
218 ```jinja
219 {% include "_common/controls.conf.j2" %}
220 {% include "_common/trusted.conf.j2" %}
221 {% include "_common/root.hint.conf" %}
222 ```
223
224 (`trusted.conf.j2` emits `trust-anchors` from the `trust_anchors` data, and
225 `root.hint.conf` points the resolver at ns1 for the root zone).
226
227 The `dnssec_py` directory is the canonical example of this pattern,
228 including several modules with different zone setups sharing one directory.
229 Declare the generated files as artifacts:
230
231 ```python
232 pytestmark = pytest.mark.extra_artifacts(
233 [
234 "ns*/dsset-*",
235 "ns*/trusted.conf",
236 "ns*/zones/*.db",
237 "ns*/zones/*.db.signed",
238 ]
239 )
240 ```
241
242
243 ## Drive named and watch its logs
244
245 The `NamedInstance` fixtures (`ns1` ... `ns11`) control the running servers.
246 The cardinal rule: never `time.sleep()` waiting for the server watch the
247 log for the line that proves the event happened.
248
249 Wait for a log line caused by an action enter the watcher *before*
250 triggering the action, so the line cannot slip past unobserved:
251
252 ```python
253 def test_cache_flush(ns4):
254 with ns4.watch_log_from_here() as watcher:
255 ns4.rndc("flush")
256 watcher.wait_for_line("flushing caches in all views succeeded")
257 ```
258
259 `wait_for_line()` accepts a string, a compiled regex, or a list of either
260 (any match wins) and returns the `re.Match`. `wait_for_all([...])` waits
261 until every pattern has appeared, `wait_for_sequence([...])` requires them
262 in order. To match lines that may already have been logged (e.g. during
263 startup), use `watch_log_from_start()` instead.
264
265 Reconfigure a server mid-test by re-rendering its config template with new
266 data, then reloading. `reconfigure()` and `reload()` send the rndc command
267 and wait for the corresponding completion line in the log:
268
269 ```python
270 def test_with_new_config(ns1, templates):
271 templates.render("ns1/named.conf", {"flag": True})
272 ns1.reconfigure()
273 ```
274
275 Dynamic updates go through `nsupdate`:
276
277 ```python
278 import dns.update
279
280 def test_update(ns1):
281 update = dns.update.UpdateMessage("example.")
282 update.add("added.example.", 300, "A", "10.0.0.2")
283 ns1.nsupdate(update)
284 ```
285
286 Send queries and check the responses with `isctest.query` and
287 `isctest.check`:
288
289 ```python
290 msg = isctest.query.create("a.example.", "A")
291 response = isctest.query.udp(msg, ns1.ip) # or .tcp(), .tls()
292 isctest.check.noerror(response) # rcode checks: nxdomain,
293 # servfail, refused, ...
294 isctest.check.adflag(response) # flag checks: rdflag,
295 # noraflag, ...
296 isctest.check.same_answer(response, expected) # compare two responses
297 ```
298
299 To wait until a zone transfer has happened, poll the SOA serial:
300
301 ```python
302 isctest.query.wait_for_serial(ns2.ip, "example.", expected_serial)
303 ```
304
305 For grepping a file the server produced (dumps, statistics), use
306 `isctest.text`:
307
308 ```python
309 from re import compile as Re
310
311 ns4.rndc("dumpdb -all")
312 dump = isctest.text.TextFile("ns4/named_dump.db")
313 assert dump.grep(Re(r"^a\.example\..*10\.0\.0\.1"))
314 ```
315