Configuration Framework

Zeek includes a configuration framework that allows updating script options at runtime. This functionality consists of an option declaration in the Zeek language, configuration files that enable changing the value of options at runtime, option-change callbacks to process updates in your Zeek scripts, a couple of script-level functions to manage config settings directly, and a log file (config.log) that contains information about every option value change according to Config::Info.

Introduction

The configuration framework provides an alternative to using Zeek script constants to store various Zeek settings.

While traditional constants work well when a value is not expected to change at runtime, they cannot be used for values that need to be modified occasionally. While a redef allows a re-definition of an already defined constant in Zeek, these redefinitions can only be performed when Zeek first starts. Afterwards, constants can no longer be modified.

However, it is clearly desirable to be able to change at runtime many of the configuration options that Zeek offers. Restarting Zeek can be time-consuming and causes it to lose all connection state and knowledge that it accumulated. Zeek’s configuration framework solves this problem.

Declaring Options

The option keyword allows variables to be declared as configuration options:

module Test;

export {
    option my_networks: set[subnet] = {};
    option enable_feature = F;
    option hostname = "testsystem";
    option timeout_after = 1min;
    option my_ports: vector of port = {};
}

Options combine aspects of global variables and constants. Like global variables, options cannot be declared inside a function, hook, or event handler. Like constants, options must be initialized when declared (the type can often be inferred from the initializer but may need to be specified when ambiguous). The value of an option can change at runtime, but options cannot be assigned a new value using normal assignments.

The initial value of an option can be redefined with a redef declaration just like for global variables and constants. However, there is no need to specify the &redef attribute in the declaration of an option. For example, given the above option declarations, here are possible redefs that work anyway:

redef Test::enable_feature = T;
redef Test::my_networks += { 10.1.0.0/16, 10.2.0.0/16 };

Changing Options

The configuration framework facilitates reading in new option values from external files at runtime. Configuration files contain a mapping between option names and their values. Each line contains one option assignment, formatted as follows:

[option name][tab/spaces][new value]

Lines starting with # are comments and ignored.

You register configuration files by adding them to Config::config_files, a set of filenames. Simply say something like the following in local.zeek:

redef Config::config_files += { "/path/to/config.dat" };

Zeek will then monitor the specified file continuously for changes. For example, editing a line containing:

Test::enable_feature T

to the config file while Zeek is running will cause it to automatically update the option’s value in the scripting layer. The next time your code accesses the option, it will see the new value.

Note

The config framework is clusterized. In a cluster configuration, only the manager node watches the specified configuration files, and relays option updates across the cluster.

Config File Formatting

The formatting of config option values in the config file is not the same as in Zeek’s scripting language. Keep an eye on the reporter.log for warnings from the config reader in case of incorrectly formatted values, which it’ll generally ignore when encountered. The following table summarizes supported types and their value representations:

Data Type

Sample Config File Entry

Comments

addr

1.2.3.4

Plain IPv4 or IPv6 address, as in Zeek. No /32 or similar netmasks.

bool

T

T or 1 for true, F or 0 for false

count

42

Plain, nonnegative integer.

double

-42.5

Plain double number.

enum

Enum::FOO_A

Plain enum string.

int

-1

Plain integer.

interval

3600.0

Always in epoch seconds, with optional fraction of seconds. Never includes a time unit.

pattern

/(foo|bar)/

The regex pattern, within forward-slash characters.

port

42/tcp

Port number with protocol, as in Zeek. When the protocol part is missing, Zeek interprets it as /unknown.

set

80/tcp,53/udp

The set members, formatted as per their own type, separated by commas. For an empty set, use an empty string: just follow the option name with whitespace.

Sets with multiple index types (e.g. set[addr,string]) are currently not supported in config files.

string

Don’t bite, Zeek

Plain string, no quotation marks. Given quotation marks become part of the string. Everything after the whitespace separator delineating the option name becomes the string. Saces and special characters are fine. Backslash characters (e.g. \n) have no special meaning.

subnet

1.2.3.4/16

Plain subnet, as in Zeek.

time

1608164505.5

Always in epoch seconds, with optional fraction of seconds. Never includes a time unit.

vector

1,2,3,4

The set members, formatted as per their own type, separated by commas. For an empty vector, use an empty string: just follow the option name with whitespace.

This leaves a few data types unsupported, notably tables and records. If you require these, build up an instance of the corresponding type manually (perhaps from a separate input framework file) and then call Config::set_value to update the option:

module Test;

export {
    option host_port: table[addr] of port = {};
}

event zeek_init() {
    local t: table[addr] of port = { [10.0.0.2] = 123/tcp };
    Config::set_value("Test::host_port", t);
}

Regardless of whether an option change is triggered by a config file or via explicit Config::set_value calls, Zeek always logs the change to config.log. A sample entry:

#fields ts      id      old_value       new_value       location
#types  time    string  string  string  string
1608167352.498872      Test::a_count     42      3      config.txt

Mentioning options repeatedly in the config files leads to multiple update events; the last entry “wins”. Mentioning options that do not correspond to existing options in the script layer is safe, but triggers warnings in reporter.log:

warning: config.txt/Input::READER_CONFIG: Option 'an_unknown' does not exist. Ignoring line.

Internally, the framework uses the Zeek input framework to learn about config changes. If you inspect the configuration framework scripts, you will notice that the scripts simply catch input framework events and call Config::set_value to set the relevant option to the new value. If you want to change an option in your scripts at runtime, you can likewise call Config::set_value directly from a script (in a cluster configuration, this only needs to happen on the manager, as the change will be automatically sent to all other nodes in the cluster).

Note

The input framework is usually very strict about the syntax of input files, but that is not the case for configuration files. These require no header lines, and both tabs and spaces are accepted as separators. A custom input reader, specifically for reading config files, facilitates this.

Tip

The gory details of option-parsing reside in Ascii::ParseValue() in src/threading/formatters/Ascii.cc and Value::ValueToVal in src/threading/SerialTypes.cc in the Zeek core.

Change Handlers

A change handler is a user-defined function that Zeek calls each time an option value changes. This allows you to react programmatically to option changes. The following example shows how to register a change handler for an option that has a data type of addr (for other data types, the return type and second parameter data type must be adjusted accordingly):

module Test;

export {
    option testaddr = 127.0.0.1;
}

# Note: the data type of 2nd parameter and return type must match
function change_addr(id: string, new_value: addr): addr
    {
    print fmt("Value of %s changed from %s to %s", id, testaddr, new_value);
    return new_value;
    }

event zeek_init()
    {
    Option::set_change_handler("Test::testaddr", change_addr);
    }

Immediately before Zeek changes the specified option value, it invokes any registered change handlers. The value returned by the change handler is the value Zeek assigns to the option. This allows, for example, checking of values to reject invalid input (the original value can be returned to override the change).

Note

Option::set_change_handler expects the name of the option to invoke the change handler for, not the option itself. Also, that name includes the module name, even when registering from within the module.

It is possible to define multiple change handlers for a single option. In this case, the change handlers are chained together: the value returned by the first change handler is the “new value” seen by the next change handler, and so on. The built-in function Option::set_change_handler takes an optional third argument that can specify a priority for the handlers.

A change handler function can optionally have a third argument of type string. When a config file triggers a change, then the third argument is the pathname of the config file. When the Config::set_value function triggers a change, then the third argument of the change handler is the value passed to the optional third argument of the Config::set_value function.

Tip

Change handlers are also used internally by the configuration framework. If you look at the script-level source code of the config framework, you can see that change handlers log the option changes to config.log.

When Change Handlers Trigger

Change handlers often implement logic that manages additional internal state. For example, depending on a performance toggle option, you might initialize or clean up a caching structure. In such scenarios you need to know exactly when and whether a handler gets invoked. The following hold:

  • When no config files get registered in Config::config_files, change handlers do not run.

  • When none of any registered config files exist on disk, change handlers do not run.

That is, change handlers are tied to config files, and don’t automatically run with the option’s default values.

  • When a config file exists on disk at Zeek startup, change handlers run with the file’s config values.

  • When the config file contains the same value the option already defaults to, its change handlers are invoked anyway.

  • zeek_init handlers run before any change handlers — i.e., they run with the options’ default values.

  • Since the config framework relies on the input framework, the input framework’s inherent asynchrony applies: you can’t assume when exactly an option change manifests in the code.

If your change handler needs to run consistently at startup and when options change, you can call the handler manually from zeek_init when you register it. That way, initialization code always runs for the option’s default value, and also for any new values.

module Test;

export {
    option use_cache = T;
}

function use_cache_hdlr(id: string, new_value: bool): bool
    {
    if ( new_value ) {
        # Ensure caching structures are set up properly
    }

    return new_value;
    }

event zeek_init()
    {
    use_cache_hdlr("Test::use_cache", use_cache);
    Option::set_change_handler("Test::use_cache", use_cache_hdlr);
    }