ZeekControl
ZeekControl (or zeekctl) is Zeek’s primary tool for cluster
orchestration. A Zeek cluster analyzes network traffic by using multiple
coordinated Zeek processes. Every Zeek process in a cluster
is called a “node”. Nodes within a cluster have different roles, such
as packet analysis, runtime state management, or logging. Each node
learns its role by assigning it a type. You use zeekctl in
order to define how many nodes of each type to run, adjust configurable
options, start (or stop) them, and check in on the running cluster.
This section will go more in depth about how to manage a Zeek cluster
using zeekctl, but it is not a reference. See the ZeekControl
documentation for more information. Since this section is entirely
about invoking zeekctl, it will not go over certain parts of the
Zeek cluster. See Zeek Cluster Setup for that.
Invoking zeekctl
When invoked by itself, zeekctl drops you into an interactive
prompt:
# zeekctl
Hint: Run the zeekctl "deploy" command to get started.
Welcome to ZeekControl 2.6.0-28
Type "help" for help.
[ZeekControl] >
Any command for zeekctl can be run within this interactive prompt or
via arguments to the zeekctl executable. We will simply execute
commands as arguments to zeekctl directly, but feel free to use the
interactive prompt if that suits you better.
Zeek ships with a default configuration for a very minimal “cluster”, using just a
single node that sniffs on the eth0 interface.
The most common zeekctl command is deploy, which launches this cluster:
# zeekctl deploy
checking configurations ...
installing ...
creating policy directories ...
installing site policies ...
generating standalone-layout.zeek ...
generating local-networks.zeek ...
generating zeekctl-config.zeek ...
generating zeekctl-config.sh ...
stopping ...
stopping zeek ...
starting ...
starting zeek ...
This does more than bring up your cluster. First, it checks for syntax errors in policy scripts. Then, it distributes configuration files to all individual systems so that they are visible to all nodes of the cluster. Finally, it orderly restarts all processes in the cluster.
This is equivalent to running three separate commands: check,
install, and restart:
# zeekctl check
zeek scripts are ok.
# zeekctl install
removing old policies in /usr/local/zeek/spool/installed-scripts-do-not-touch/site ...
removing old policies in /usr/local/zeek/spool/installed-scripts-do-not-touch/auto ...
creating policy directories ...
installing site policies ...
generating standalone-layout.zeek ...
generating local-networks.zeek ...
generating zeekctl-config.zeek ...
generating zeekctl-config.sh ...
# zeekctl restart
stopping ...
stopping zeek ...
starting ...
starting zeek ..
But, you may as well just use deploy whenever you make changes to
any policy scripts or configuration files. Let’s swap the default
cluster out for a “real” cluster, then redeploy as you would in a
production environment.
Before you change the cluster layout, make sure you run zeekctl stop
in order to stop all of the current running nodes (which will just be
the standalone node):
# zeekctl stop
stopping zeek ...
Note
This section will use $PREFIX to refer to Zeek’s installation
directory. This is set as an environment variable by the tutorial
setup script, so you can use it verbatim.
Now, you can change the cluster setup to include multiple Zeek
processes. First, modify $PREFIX/etc/node.cfg by uncommenting the
“example clustered configuration” and commenting out the “standalone”
configuration. Here is the clustered configuration:
[logger-1]
type=logger
host=localhost
[manager]
type=manager
host=localhost
[proxy-1]
type=proxy
host=localhost
[worker-1]
type=worker
host=localhost
interface=eth0
[worker-2]
type=worker
host=localhost
interface=eth0
Since we changed the cluster, you should run zeekctl deploy and see
that the new topology is running:
# zeekctl deploy
<...>
starting ...
starting logger ...
starting manager ...
starting proxy ...
starting workers ...
Now we started the logger, manager, proxy, and workers. With a running
cluster, you can view its status in a few ways. The most interactive is
with zeekctl top:
# zeekctl top
Name Type Host Pid VSize Rss Cpu Cmd
logger-1 logger localhost 4341 2G 154M 0% zeek
manager manager localhost 4394 1013M 154M 0% zeek
proxy-1 proxy localhost 4446 1013M 153M 0% zeek
worker-1 worker localhost 4511 1G 279M 0% zeek
worker-2 worker localhost 4512 1G 279M 0% zeek
This command is different in the “interactive” mode of zeekctl—try
running zeekctl alone, then top. Here you can watch the various
process resources live!
You can also check the status of your Zeek cluster with zeekctl
status:
# zeekctl status
Name Type Host Status Pid Started
logger-1 logger localhost running 4341 08 Dec 21:47:32
manager manager localhost running 4394 08 Dec 21:47:33
proxy-1 proxy localhost running 4446 08 Dec 21:47:34
worker-1 worker localhost running 4511 08 Dec 21:47:35
worker-2 worker localhost running 4512 08 Dec 21:47:35
This is useful to ensure the nodes are healthy and running.
If anything is wrong, you can also use zeekctl diag on the node. For
example, here we force the proxy-1 node to immediately crash, then
check its stderr.log with zeekctl diag:
# echo "@if ( cluster::node == \"proxy-1\" ) event zeek_init() { Reporter::fatal(\"bad\!\"); } @endif" >> $PREFIX/share/zeek/site/local.zeek
# zeekctl deploy
<...>
starting proxy ...
Error: proxy-1 terminated immediately after starting; check output with "diag"
# zeekctl diag proxy-1
<...>
==== stderr.log
fatal error in /usr/local/zeek/spool/installed-scripts-do-not-touch/site/local.zeek, line 1: bad!
<...>
# sed -i '$d' $PREFIX/share/zeek/site/local.zeek # Remove the line we just added
The diag command is essential to diagnose issues with the cluster.
If you see that a certain node is stopped via zeekctl status and
need to restart just that node, you can specify it as an argument. For
example, here we kill the proxy-1 node, then start just that node
after killing it:
# kill 4446
# zeekctl status
Name Type Host Status Pid Started
logger-1 logger localhost running 4341 08 Dec 21:47:32
manager manager localhost running 4394 08 Dec 21:47:33
proxy-1 proxy localhost crashed
worker-1 worker localhost running 4511 08 Dec 21:47:35
worker-2 worker localhost running 4512 08 Dec 21:47:35
# zeekctl start proxy-1
starting proxy ...
# zeekctl status
Name Type Host Status Pid Started
logger-1 logger localhost running 4341 08 Dec 21:47:32
manager manager localhost running 4394 08 Dec 21:47:33
proxy-1 proxy localhost running 4947 08 Dec 21:53:49
worker-1 worker localhost running 4511 08 Dec 21:47:35
worker-2 worker localhost running 4512 08 Dec 21:47:35
This required manual intervention. Instead, you can use zeekctl
with cron in order to automatically check for crashed nodes and
restart them. See the ZeekControl cron command reference for more
information.
Note
ZeekControl itself has no active process monitoring. Therefore, in
order to perform automatic restarts, you need to put the zeekctl
cron command into a crontab entry.
Customizing Zeek in a Cluster Setting
In the above we intentionally introduced an error to a Zeek script called
$PREFIX/share/zeek/site/local.zeek. This file has a special purpose: it is
your entry point to persistent Zeek customizations, such as the FTP adjustments
we made earlier on the command line. We suggest you take a minute to poke
through it! Among other suggested settings, it includes a (commented-out) line
to load any installed Zeek packages. Feel free to experiment with changes to the
configuration, and deploying them via zeekctl deploy.
Many basic clusters can be maintained with just what was discussed here:
a mixture of zeekctl commands like deploy, start, stop,
and status. You can use top and cron for extra visibility or
monitoring on your cluster. Then, configure the cluster according to
your particular environment, and that should be what you need to get a
capable Zeek cluster.
In the next section, we will go over analyzing logs. If you just want to see how logs function when using ZeekControl, see the Analyzing Logs from Zeekctl section.