xref: /src/crypto/external/apache2/openssl/dist/test/radix

RADIX Test Framework
====================

Purpose
-------

This directory contains the RADIX test framework, which is a six-dimension
script-driven facility intended to facilitate execution of

- multi-stream
- multi-client
- multi-server
- multi-thread
- multi-process (in future)
- multi-node (in future)

test vignettes for network protocol testing applications. While it is currently
used for QUIC, it has been designed to be agnostic so that it can be adapted to
other protocols in future if desired.

In particular, unlike the older multistream test framework, it does not assume a
single client and a single server. Examples of vignettes designed to be
supported by the RADIX test framework in future include:

- single client  single server
- multiple clients  single server
- single client  multiple servers
- multiple clients  multiple servers

Multi-process and multi-node means there has been some consideration
given to support of multi-process and multi-node testing in the future, though
this is not currently supported.

Differences to `quic_multistream_test`
--------------------------------------

The RADIX test features the following improvements relative to the
`quic_multistream_test` framework:

- Due to usage of the new QUIC server API, opcodes are no longer duplicated
  into C and S variants. There is symmetry between all endpoints in this
  regard. The legacy `QUIC_TSERVER` facility is not used and will eventually
  be retired once other test suites no longer rely on it.

- The framework is not limited to two objects (client and server),
  but can instead instantiate an arbitrary number of SSL objects and pass these
  to operations. Other kinds of object could be supported in future if needed.

- Scripts are now generated dynamically at launch time by functions rather
  than hardcoding them into the executable. This has the advantage that scripts
  can be generated dynamically. In particular, this allows very long
  procedurally generated test scripts which would be impractical to write
  by hand. This can be used for stress testing, for example.

- As a result of the fact that scripts are now generated dynamically, the
  in-memory representation of script operations has been improved as the ability
  to write a script operation as an initializer list is no longer required. The
  new representation is simpler, more compact, and more flexible. A stack-based
  approach allows arbitrary argument types to be passed as operands to an
  operation, rather than having a fixed number of operands of fixed type for all
  script operations.

- Scripts are now named, rather than numbered, giving more useful debug output,
  and hopefully reducing the frequency of merge conflicts.

- Debug logging has in general been significantly improved and has been designed
  to be accessible and useful.

- Logging of child threads is now buffered and printed after a test is complete,
  which avoids non-deterministic interleaving of test output.

- The number of core opcodes for the interpreter has been dramatically reduced
  and now the vast majority of test operations are performed via `OP_FUNC`,
  which is similar to `OP_CHECK`. This change is largely transparent to the
  test developer.

- In the future, multi-process or multi-node testing will be supported.
  The expectation is that multi-node testing will be facilitated by having the
  master process invoke a hook shell script which is responsible for propping up
  and tearing down the additional nodes. Writing a suitable hook script will be
  left as an exercise to the test infrastructure maintainer. This abstracts the
  test framework from the details of a specific infrastructure environment and
  its management tools (VMs, containers, etc.).

- The core test interpreter is designed to be agnostic to QUIC and could be
  used for testing other protocols in the future. There is a clean, layered
  design which draws a clear distinction between the core interpreter,
  protocol-specific bindings and support code, test operation definitions, and
  script definitions.

- It is no longer needed to explicitly set the ALPN using an opcode when
  establishing a connection using QUIC. Since ALPN is required for QUIC, the
  first opcode of every test was used to set the ALPN to the same string, which
  was redundant. This is now done automatically.

- An explicit `OP_END` is no longer needed.

Architecture
------------

The RADIX test suite framework is built in four layers:

- **TERP** ([terp.c](./terp.c)), a protocol-agnostic stack-based script
  interpreter for interruptible execution of test vignettes;

- the **QUIC bindings** ([quic_bindings.c](./quic_bindings.c)), which defines
  QUIC-specific test framework;

- the **QUIC operations** ([quic_ops.c](./quic_ops.c)), which define specific
  test operations for TERP which can be invoked by QUIC unit tests on top of the
  basic infrastructure defined by the QUIC bindings;

- the QUIC unit tests ([quic_tests.c](./quic_tests.c)), which use the above
  QUIC bindings.