base/frameworks/cluster/main.zeek

Cluster

A framework for establishing and controlling a cluster of Zeek instances. In order to use the cluster framework, a script named cluster-layout.zeek must exist somewhere in Zeek’s script search path which has a cluster definition of the Cluster::nodes variable. The CLUSTER_NODE environment variable or Cluster::node must also be sent and the cluster framework loaded as a package like @load base/frameworks/cluster.

Warning

The file cluster-layout.zeek should only contain the definition of Cluster::nodes. Specifically, avoid loading other Zeek scripts or using redef for anything but Cluster::nodes.

Due to cluster-layout.zeek being loaded very early, it is easy to introduce circular loading issues.

Namespace

Cluster

Imports

base/frameworks/broker, base/frameworks/control

Summary

Redefinable Options

Cluster::default_backend: Broker::BackendType &redef

The type of data store backend that will be used for all data stores if no other has already been specified by the user in Cluster::stores.

Cluster::default_master_node: string &redef

Name of the node on which master data stores will be created if no other has already been specified by the user in Cluster::stores.

Cluster::default_persistent_backend: Broker::BackendType &redef

The type of persistent data store backend that will be used for all data stores if no other has already been specified by the user in Cluster::stores.

Cluster::default_store_dir: string &redef

Setting a default dir will, for persistent backends that have not been given an explicit file path via Cluster::stores, automatically create a path within this dir that is based on the name of the data store.

Cluster::enable_round_robin_logging: bool &redef

Whether to distribute log messages among available logging nodes.

Cluster::logger_topic: string &redef

The topic name used for exchanging messages that are relevant to logger nodes in a cluster.

Cluster::manager_is_logger: bool &redef

Indicates whether or not the manager will act as the logger and receive logs.

Cluster::manager_topic: string &redef

The topic name used for exchanging messages that are relevant to manager nodes in a cluster.

Cluster::node: string &redef

This is usually supplied on the command line for each instance of the cluster that is started up.

Cluster::node_topic_prefix: string &redef

The topic prefix used for exchanging messages that are relevant to a named node in a cluster.

Cluster::nodeid_topic_prefix: string &redef

The topic prefix used for exchanging messages that are relevant to a unique node in a cluster.

Cluster::nodes: table &redef

The cluster layout definition.

Cluster::proxy_topic: string &redef

The topic name used for exchanging messages that are relevant to proxy nodes in a cluster.

Cluster::retry_interval: interval &redef

Interval for retrying failed connections between cluster nodes.

Cluster::time_machine_topic: string &redef

The topic name used for exchanging messages that are relevant to time machine nodes in a cluster.

Cluster::worker_topic: string &redef

The topic name used for exchanging messages that are relevant to worker nodes in a cluster.

Constants

Cluster::broadcast_topics: set

A set of topic names to be used for broadcasting messages that are relevant to all nodes in a cluster.

State Variables

Cluster::stores: table &default = &redef

A table of cluster-enabled data stores that have been created, indexed by their name.

Cluster::worker_count: count &deprecated =

This gives the value for the number of workers currently connected to, and it’s maintained internally by the cluster framework.

Types

Cluster::Info: record &log

The record type which contains the column fields of the cluster log.

Cluster::NamedNode: record

Record to represent a cluster node including its name.

Cluster::Node: record

Record type to indicate a node in a cluster.

Cluster::NodeType: enum

Types of nodes that are allowed to participate in the cluster configuration.

Cluster::StoreInfo: record

Information regarding a cluster-enabled data store.

Redefinitions

Log::ID: enum

The cluster logging stream identifier.

Events

Cluster::hello: event

When using broker-enabled cluster framework, nodes broadcast this event to exchange their user-defined name along with a string that uniquely identifies it for the duration of its lifetime.

Cluster::node_down: event

When using broker-enabled cluster framework, this event will be emitted locally whenever a connected cluster node becomes disconnected.

Cluster::node_up: event

When using broker-enabled cluster framework, this event will be emitted locally whenever a cluster node connects or reconnects.

Hooks

Cluster::log_policy: Log::PolicyHook

A default logging policy hook for the stream.

Functions

Cluster::create_store: function

Sets up a cluster-enabled data store.

Cluster::get_active_node_count: function

Returns the number of nodes per type, the calling node is currently connected to.

Cluster::get_node_count: function

Returns the number of nodes defined in the cluster layout for a given node type.

Cluster::is_enabled: function

This function can be called at any time to determine if the cluster framework is being enabled for this run.

Cluster::local_node_type: function

This function can be called at any time to determine what type of cluster node the current Zeek instance is going to be acting as.

Cluster::log: function

Write a message to the cluster logging stream.

Cluster::node_topic: function

Retrieve the topic associated with a specific node in the cluster.

Cluster::nodeid_topic: function

Retrieve the topic associated with a specific node in the cluster.

Detailed Interface

Redefinable Options

Cluster::default_backend
Type

Broker::BackendType

Attributes

&redef

Default

Broker::MEMORY

The type of data store backend that will be used for all data stores if no other has already been specified by the user in Cluster::stores.

Cluster::default_master_node
Type

string

Attributes

&redef

Default

""

Name of the node on which master data stores will be created if no other has already been specified by the user in Cluster::stores. An empty value means “use whatever name corresponds to the manager node”.

Cluster::default_persistent_backend
Type

Broker::BackendType

Attributes

&redef

Default

Broker::SQLITE

The type of persistent data store backend that will be used for all data stores if no other has already been specified by the user in Cluster::stores. This will be used when script authors call Cluster::create_store with the persistent argument set true.

Cluster::default_store_dir
Type

string

Attributes

&redef

Default

""

Setting a default dir will, for persistent backends that have not been given an explicit file path via Cluster::stores, automatically create a path within this dir that is based on the name of the data store.

Cluster::enable_round_robin_logging
Type

bool

Attributes

&redef

Default

T

Whether to distribute log messages among available logging nodes.

Cluster::logger_topic
Type

string

Attributes

&redef

Default

"zeek/cluster/logger"

The topic name used for exchanging messages that are relevant to logger nodes in a cluster. Used with broker-enabled cluster communication.

Cluster::manager_is_logger
Type

bool

Attributes

&redef

Default

T

Indicates whether or not the manager will act as the logger and receive logs. This value should be set in the cluster-layout.zeek script (the value should be true only if no logger is specified in Cluster::nodes). Note that ZeekControl handles this automatically.

Cluster::manager_topic
Type

string

Attributes

&redef

Default

"zeek/cluster/manager"

The topic name used for exchanging messages that are relevant to manager nodes in a cluster. Used with broker-enabled cluster communication.

Cluster::node
Type

string

Attributes

&redef

Default

""

This is usually supplied on the command line for each instance of the cluster that is started up.

Cluster::node_topic_prefix
Type

string

Attributes

&redef

Default

"zeek/cluster/node/"

The topic prefix used for exchanging messages that are relevant to a named node in a cluster. Used with broker-enabled cluster communication.

Cluster::nodeid_topic_prefix
Type

string

Attributes

&redef

Default

"zeek/cluster/nodeid/"

The topic prefix used for exchanging messages that are relevant to a unique node in a cluster. Used with broker-enabled cluster communication.

Cluster::nodes
Type

table [string] of Cluster::Node

Attributes

&redef

Default

{}

The cluster layout definition. This should be placed into a filter named cluster-layout.zeek somewhere in the ZEEKPATH. It will be automatically loaded if the CLUSTER_NODE environment variable is set. Note that ZeekControl handles all of this automatically. The table is typically indexed by node names/labels (e.g. “manager” or “worker-1”).

Cluster::proxy_topic
Type

string

Attributes

&redef

Default

"zeek/cluster/proxy"

The topic name used for exchanging messages that are relevant to proxy nodes in a cluster. Used with broker-enabled cluster communication.

Cluster::retry_interval
Type

interval

Attributes

&redef

Default

1.0 min

Interval for retrying failed connections between cluster nodes. If set, the ZEEK_DEFAULT_CONNECT_RETRY (given in number of seconds) environment variable overrides this option.

Cluster::time_machine_topic
Type

string

Attributes

&redef

Default

"zeek/cluster/time_machine"

The topic name used for exchanging messages that are relevant to time machine nodes in a cluster. Used with broker-enabled cluster communication.

Cluster::worker_topic
Type

string

Attributes

&redef

Default

"zeek/cluster/worker"

The topic name used for exchanging messages that are relevant to worker nodes in a cluster. Used with broker-enabled cluster communication.

Constants

Cluster::broadcast_topics
Type

set [string]

Default
{
   "zeek/cluster/manager",
   "zeek/cluster/logger",
   "zeek/cluster/proxy",
   "zeek/cluster/worker",
   "zeek/cluster/time_machine"
}

A set of topic names to be used for broadcasting messages that are relevant to all nodes in a cluster. Currently, there is not a common topic to broadcast to, because enabling implicit Broker forwarding would cause a routing loop for this topic.

State Variables

Cluster::stores
Type

table [string] of Cluster::StoreInfo

Attributes

&default = [name=<uninitialized>, store=<uninitialized>, master_node=, master=F, backend=Broker::MEMORY, options=[sqlite=[path=, synchronous=<uninitialized>, journal_mode=<uninitialized>, failure_mode=Broker::SQLITE_FAILURE_MODE_FAIL, integrity_check=F]], clone_resync_interval=10.0 secs, clone_stale_interval=5.0 mins, clone_mutation_buffer_interval=2.0 mins] &redef

Default

{}

A table of cluster-enabled data stores that have been created, indexed by their name. This table will be populated automatically by Cluster::create_store, but if you need to customize the options related to a particular data store, you may redef this table. Calls to Cluster::create_store will first check the table for an entry of the same name and, if found, will use the predefined options there when setting up the store.

Cluster::worker_count
Type

count

Attributes

&deprecated = “Remove in v6.1. Active worker count can be obtained via get_active_node_count(Cluster::WORKER)”

Default

0

This gives the value for the number of workers currently connected to, and it’s maintained internally by the cluster framework. It’s primarily intended for use by managers to find out how many workers should be responding to requests.

Types

Cluster::Info
Type

record

ts: time &log

The time at which a cluster message was generated.

node: string &log

The name of the node that is creating the log record.

message: string &log

A message indicating information about the cluster’s operation.

Attributes

&log

The record type which contains the column fields of the cluster log.

Cluster::NamedNode
Type

record

name: string

node: Cluster::Node

Record to represent a cluster node including its name.

Cluster::Node
Type

record

node_type: Cluster::NodeType

Identifies the type of cluster node in this node’s configuration.

ip: addr

The IP address of the cluster node.

zone_id: string &default = "" &optional

If the ip field is a non-global IPv6 address, this field can specify a particular RFC 4007 zone_id.

p: port &default = 0/unknown &optional

The port that this node will listen on for peer connections. A value of 0/unknown means the node is not pre-configured to listen.

interface: string &optional

Identifier for the interface a worker is sniffing.

manager: string &optional

Name of the manager node this node uses. For workers and proxies.

time_machine: string &optional

Name of a time machine node with which this node connects.

id: string &optional

A unique identifier assigned to the node by the broker framework. This field is only set while a node is connected.

lb_filter: string &optional

(present if policy/misc/load-balancing.zeek is loaded)

A BPF filter for load balancing traffic sniffed on a single interface across a number of processes. In normal uses, this will be assigned dynamically by the manager and installed by the workers.

Record type to indicate a node in a cluster.

Cluster::NodeType
Type

enum

Cluster::NONE

A dummy node type indicating the local node is not operating within a cluster.

Cluster::CONTROL

A node type which is allowed to view/manipulate the configuration of other nodes in the cluster.

Cluster::LOGGER

A node type responsible for log management.

Cluster::MANAGER

A node type responsible for policy management.

Cluster::PROXY

A node type for relaying worker node communication and synchronizing worker node state.

Cluster::WORKER

The node type doing all the actual traffic analysis.

Cluster::TIME_MACHINE

A node acting as a traffic recorder using the Time Machine software.

Types of nodes that are allowed to participate in the cluster configuration.

Cluster::StoreInfo
Type

record

name: string &optional

The name of the data store.

store: opaque of Broker::Store &optional

The store handle.

master_node: string &default = Cluster::default_master_node &optional

The name of the cluster node on which the master version of the data store resides.

master: bool &default = F &optional

Whether the data store is the master version or a clone.

backend: Broker::BackendType &default = Cluster::default_backend &optional

The type of backend used for storing data.

options: Broker::BackendOptions &default = [sqlite=[path=, synchronous=<uninitialized>, journal_mode=<uninitialized>, failure_mode=Broker::SQLITE_FAILURE_MODE_FAIL, integrity_check=F]] &optional

Parameters used for configuring the backend.

clone_resync_interval: interval &default = Broker::default_clone_resync_interval &optional

A resync/reconnect interval to pass through to Broker::create_clone.

clone_stale_interval: interval &default = Broker::default_clone_stale_interval &optional

A staleness duration to pass through to Broker::create_clone.

clone_mutation_buffer_interval: interval &default = Broker::default_clone_mutation_buffer_interval &optional

A mutation buffer interval to pass through to Broker::create_clone.

Information regarding a cluster-enabled data store.

Events

Cluster::hello
Type

event (name: string, id: string)

When using broker-enabled cluster framework, nodes broadcast this event to exchange their user-defined name along with a string that uniquely identifies it for the duration of its lifetime. This string may change if the node dies and has to reconnect later.

Cluster::node_down
Type

event (name: string, id: string)

When using broker-enabled cluster framework, this event will be emitted locally whenever a connected cluster node becomes disconnected.

Cluster::node_up
Type

event (name: string, id: string)

When using broker-enabled cluster framework, this event will be emitted locally whenever a cluster node connects or reconnects.

Hooks

Cluster::log_policy
Type

Log::PolicyHook

A default logging policy hook for the stream.

Functions

Cluster::create_store
Type

function (name: string, persistent: bool &default = F &optional) : Cluster::StoreInfo

Sets up a cluster-enabled data store. They will also still properly function for uses that are not operating a cluster.

Parameters
  • name – the name of the data store to create.

  • persistent – whether the data store must be persistent.

Returns

the store’s information. For master stores, the store will be ready to use immediately. For clones, the store field will not be set until the node containing the master store has connected.

Cluster::get_active_node_count
Type

function (node_type: Cluster::NodeType) : count

Returns the number of nodes per type, the calling node is currently connected to. This is primarily intended for use by the manager to find out how many nodes should be responding to requests.

Cluster::get_node_count
Type

function (node_type: Cluster::NodeType) : count

Returns the number of nodes defined in the cluster layout for a given node type.

Cluster::is_enabled
Type

function () : bool

This function can be called at any time to determine if the cluster framework is being enabled for this run.

Returns

True if Cluster::node has been set.

Cluster::local_node_type
Type

function () : Cluster::NodeType

This function can be called at any time to determine what type of cluster node the current Zeek instance is going to be acting as. If Cluster::is_enabled returns false, then Cluster::NONE is returned.

Returns

The Cluster::NodeType the calling node acts as.

Cluster::log
Type

function (msg: string) : void

Write a message to the cluster logging stream.

Cluster::node_topic
Type

function (name: string) : string

Retrieve the topic associated with a specific node in the cluster.

Parameters

name – the name of the cluster node (e.g. “manager”).

Returns

a topic string that may used to send a message exclusively to a given cluster node.

Cluster::nodeid_topic
Type

function (id: string) : string

Retrieve the topic associated with a specific node in the cluster.

Parameters

id – the id of the cluster node (from Broker::EndpointInfo or Broker::node_id.

Returns

a topic string that may used to send a message exclusively to a given cluster node.