10.2. Testing

Spicy’s testing & CI setup includes several pieces that we discuss in the following.

TLDR; If you make changes, make sure that make check runs through. You need the right clang-format (see Formatting) and clang-tidy (see Static analysis) versions for that (from Clang >=10). If you don’t have them (or want to save time), run at least make test. If that still takes too long for you, run make test-core.

10.2.1. BTest

Most tests are end-to-end tests that work from Spicy (or HILTI) source code and check that everything compiles and produces the expected output. We use BTest to drive these, very similar to Zeek. make test from the top-level directory will execute these tests. You get the same effect by changing into tests/ and running btest -j there (-j parallelizes test execution).

The most important BTest options are:

  • -d prints debugging output for failing tests to the console

  • -f diag.log records the same debugging output into diag.log

  • -u updates baselines when output changes in expected ways (don’t forget to commit the updates)

There are some alternatives to running just all tests, per the following:

Running tests using installation after make install

By default, btests are running completely out of the source & build directories. If you run btest -a installation, BTest will instead switch to pulling everything from their installation locations. If you have already deleted the build directory, you also need to have the environment variable SPICY_INSTALLATION_DIRECTORY point to your installation prefix, as otherwise BTest has no way of knowing where to find Spicy.

10.2.2. Unit tests

There’s a growing set of units test. These are using doctest and are executed through btest as well, so just running tests per above will have these included.

Alternatively, the test binaries in the build directory can be executed to exercise the tests, or one can use the check build target to execute all unit tests.

10.2.3. Sanitizers

To build tools and libraries with support for Clang’s address/leak sanitizer, configure with --enable-sanitizer. If Clang’s asan libraries aren’t in a standard runtime library path, you’ll also need to set LD_LIBRARY_PATH (Linux) or DYLD_LIBRARY_PATH (macOS) to point there (e.g., LD_LIBRARY_PATH=/opt/clang9/lib/clang/9.0.1/lib/linux).

When using the Spicy plugin for Zeek and Zeek hasn’t been compiled with sanitizer support, you’ll also need to set LD_PRELOAD (Linux) or DYLD_INSERT_LIBRARIES (macOS) to the shared asan library to use (e.g., LD_PRELOAD=/data/clang9/lib/clang/9.0.1/lib/linux/libclang_rt.asan-x86_64.so). Because you probably don’t want to set that permanently, the test suite pays attention to a variable ZEEK_LD_PRELOAD: If you set that before running btest to the path you want in LD_PRELOAD, the relevant tests will copy the value for running Zeek.

To make the sanitizer symbolize its output you need to set the ASAN_SYMBOLIZER_PATH environment variable to point to the llvm-symbolizer binary, or make sure llvm-symbolizer is in your PATH.

Note

As we are running one of the CI build with sanitizers, it’s ok not to run this locally on a regular basis during development.

10.2.4. Code Quality

Our CI runs the Formatting and Static analysis checks, and will fail if any of that doesn’t pass. To execute these locally, run the make target format and tidy, respectively. Don’t forget to set CLANG_FORMAT and CLANG_TIDY to the right version of the binary if they aren’t in your PATH.

CI also runs pre-commit with a configuration pre-configured in .pre-commit-config.yaml. To run that locally on every commit, install pre-commit and then put its git hook in place through executing pre-commit install; see the installation instructions for more details.

10.2.5. Docker Builds

We are shipping a number of Docker files in docker/; see Using Docker for more information. As part of our CI, we make sure these build OK and pass btest -a installation. If you have Docker available, you can run these individually yourself through make test-<platform> in docker/. However, usually it’s fine to leave this to CI.

10.2.6. How Test Your Branch

If you run make check in the top-level directory you get the combination of all the btests, formatting, and linting. That’s the best check to do to make sure your branch is in good shape, in particular before filing a pull request.