.. _zeekcontrol cron: https://github.com/zeek/zeekctl?tab=readme-ov-file#zeekcontrol-cron-command .. _zeekcontrol documentation: https://github.com/zeek/zeekctl .. _invoking-zeekctl: ############# ZeekControl ############# ZeekControl (or ``zeekctl``) is Zeek's primary tool for cluster orchestration. A Zeek cluster analyzes network traffic by using multiple coordinated Zeek processes. Every Zeek process in a cluster is called a "node". Nodes within a cluster have different roles, such as packet analysis, runtime state management, or logging. Each node learns its role by assigning it a type. You use ``zeekctl`` in order to define how many nodes of each type to run, adjust configurable options, start (or stop) them, and check in on the running cluster. This section will go more in depth about how to manage a Zeek cluster using ``zeekctl``, but it is not a reference. See the `ZeekControl documentation`_ for more information. Since this section is entirely about invoking ``zeekctl``, it will not go over certain parts of the Zeek cluster. See :ref:`cluster-setup` for that. ====================== Invoking ``zeekctl`` ====================== When invoked by itself, ``zeekctl`` drops you into an interactive prompt: .. code:: console # zeekctl Hint: Run the zeekctl "deploy" command to get started. Welcome to ZeekControl 2.6.0-28 Type "help" for help. [ZeekControl] > Any command for ``zeekctl`` can be run within this interactive prompt or via arguments to the ``zeekctl`` executable. We will simply execute commands as arguments to ``zeekctl`` directly, but feel free to use the interactive prompt if that suits you better. Zeek ships with a default configuration for a very minimal "cluster", using just a single node that sniffs on the ``eth0`` interface. The most common ``zeekctl`` command is ``deploy``, which launches this cluster: .. code:: console # zeekctl deploy checking configurations ... installing ... creating policy directories ... installing site policies ... generating standalone-layout.zeek ... generating local-networks.zeek ... generating zeekctl-config.zeek ... generating zeekctl-config.sh ... stopping ... stopping zeek ... starting ... starting zeek ... This does more than bring up your cluster. First, it checks for syntax errors in policy scripts. Then, it distributes configuration files to all individual systems so that they are visible to all nodes of the cluster. Finally, it orderly restarts all processes in the cluster. This is equivalent to running three separate commands: ``check``, ``install``, and ``restart``: .. code:: console # zeekctl check zeek scripts are ok. # zeekctl install removing old policies in /usr/local/zeek/spool/installed-scripts-do-not-touch/site ... removing old policies in /usr/local/zeek/spool/installed-scripts-do-not-touch/auto ... creating policy directories ... installing site policies ... generating standalone-layout.zeek ... generating local-networks.zeek ... generating zeekctl-config.zeek ... generating zeekctl-config.sh ... # zeekctl restart stopping ... stopping zeek ... starting ... starting zeek .. But, you may as well just use ``deploy`` whenever you make changes to any policy scripts or configuration files. Let's swap the default cluster out for a "real" cluster, then redeploy as you would in a production environment. Before you change the cluster layout, make sure you run ``zeekctl stop`` in order to stop all of the current running nodes (which will just be the standalone node): .. code:: console # zeekctl stop stopping zeek ... .. note:: This section will use ``$PREFIX`` to refer to Zeek's installation directory. This is set as an environment variable by the tutorial setup script, so you can use it verbatim. Now, you can change the cluster setup to include multiple Zeek processes. First, modify ``$PREFIX/etc/node.cfg`` by uncommenting the "example clustered configuration" and commenting out the "standalone" configuration. Here is the clustered configuration: .. code:: console [logger-1] type=logger host=localhost [manager] type=manager host=localhost [proxy-1] type=proxy host=localhost [worker-1] type=worker host=localhost interface=eth0 [worker-2] type=worker host=localhost interface=eth0 Since we changed the cluster, you should run ``zeekctl deploy`` and see that the new topology is running: .. code:: console # zeekctl deploy <...> starting ... starting logger ... starting manager ... starting proxy ... starting workers ... Now we started the logger, manager, proxy, and workers. With a running cluster, you can view its status in a few ways. The most interactive is with ``zeekctl top``: .. code:: console # zeekctl top Name Type Host Pid VSize Rss Cpu Cmd logger-1 logger localhost 4341 2G 154M 0% zeek manager manager localhost 4394 1013M 154M 0% zeek proxy-1 proxy localhost 4446 1013M 153M 0% zeek worker-1 worker localhost 4511 1G 279M 0% zeek worker-2 worker localhost 4512 1G 279M 0% zeek This command is different in the "interactive" mode of ``zeekctl``---try running ``zeekctl`` alone, then ``top``. Here you can watch the various process resources live! You can also check the status of your Zeek cluster with ``zeekctl status``: .. code:: console # zeekctl status Name Type Host Status Pid Started logger-1 logger localhost running 4341 08 Dec 21:47:32 manager manager localhost running 4394 08 Dec 21:47:33 proxy-1 proxy localhost running 4446 08 Dec 21:47:34 worker-1 worker localhost running 4511 08 Dec 21:47:35 worker-2 worker localhost running 4512 08 Dec 21:47:35 This is useful to ensure the nodes are healthy and running. If anything is wrong, you can also use ``zeekctl diag`` on the node. For example, here we force the ``proxy-1`` node to immediately crash, then check its ``stderr.log`` with ``zeekctl diag``: .. code:: console # echo "@if ( cluster::node == \"proxy-1\" ) event zeek_init() { Reporter::fatal(\"bad\!\"); } @endif" >> $PREFIX/share/zeek/site/local.zeek # zeekctl deploy <...> starting proxy ... Error: proxy-1 terminated immediately after starting; check output with "diag" # zeekctl diag proxy-1 <...> ==== stderr.log fatal error in /usr/local/zeek/spool/installed-scripts-do-not-touch/site/local.zeek, line 1: bad! <...> # sed -i '$d' $PREFIX/share/zeek/site/local.zeek # Remove the line we just added The ``diag`` command is essential to diagnose issues with the cluster. If you see that a certain node is stopped via ``zeekctl status`` and need to restart just that node, you can specify it as an argument. For example, here we kill the ``proxy-1`` node, then start just that node after killing it: .. code:: console # kill 4446 # zeekctl status Name Type Host Status Pid Started logger-1 logger localhost running 4341 08 Dec 21:47:32 manager manager localhost running 4394 08 Dec 21:47:33 proxy-1 proxy localhost crashed worker-1 worker localhost running 4511 08 Dec 21:47:35 worker-2 worker localhost running 4512 08 Dec 21:47:35 # zeekctl start proxy-1 starting proxy ... # zeekctl status Name Type Host Status Pid Started logger-1 logger localhost running 4341 08 Dec 21:47:32 manager manager localhost running 4394 08 Dec 21:47:33 proxy-1 proxy localhost running 4947 08 Dec 21:53:49 worker-1 worker localhost running 4511 08 Dec 21:47:35 worker-2 worker localhost running 4512 08 Dec 21:47:35 This required manual intervention. Instead, you can use ``zeekctl`` with ``cron`` in order to automatically check for crashed nodes and restart them. See the `ZeekControl cron`_ command reference for more information. .. note:: ZeekControl itself has no active process monitoring. Therefore, in order to perform automatic restarts, you need to put the ``zeekctl cron`` command into a ``crontab`` entry. ======================================= Customizing Zeek in a Cluster Setting ======================================= In the above we intentionally introduced an error to a Zeek script called ``$PREFIX/share/zeek/site/local.zeek``. This file has a special purpose: it is your entry point to persistent Zeek customizations, such as the FTP adjustments we made earlier on the command line. We suggest you take a minute to poke through it! Among other suggested settings, it includes a (commented-out) line to load any installed Zeek packages. Feel free to experiment with changes to the configuration, and deploying them via ``zeekctl deploy``. Many basic clusters can be maintained with just what was discussed here: a mixture of ``zeekctl`` commands like ``deploy``, ``start``, ``stop``, and ``status``. You can use ``top`` and ``cron`` for extra visibility or monitoring on your cluster. Then, configure the cluster according to your particular environment, and that should be what you need to get a capable Zeek cluster. In the next section, we will go over analyzing logs. If you just want to see how logs function when using ZeekControl, see the :ref:`zeekcontrol_logs` section.