main.zeek

Cluster

A framework for establishing and controlling a cluster of Zeek instances. In order to use the cluster framework, a script named cluster-layout.zeek must exist somewhere in Zeek’s script search path which has a cluster definition of the Cluster::nodes variable. The CLUSTER_NODE environment variable or Cluster::node must also be sent and the cluster framework loaded as a package like @load base/frameworks/cluster.

Warning

The file cluster-layout.zeek should only contain the definition of Cluster::nodes. Specifically, avoid loading other Zeek scripts or using redef for anything but Cluster::nodes.

Due to cluster-layout.zeek being loaded very early, it is easy to introduce circular loading issues.

Namespace:: Cluster
Imports:: base/bif/cluster.bif.zeek, base/bif/plugins/Zeek_Cluster_WebSocket.events.bif.zeek, base/frameworks/broker, base/frameworks/cluster/types.zeek, base/frameworks/control

Summary

Redefinable Options

`Cluster::default_backend`: `Broker::BackendType` `&redef`	The type of data store backend that will be used for all data stores if no other has already been specified by the user in `Cluster::stores`.
`Cluster::default_master_node`: `string` `&redef`	Name of the node on which master data stores will be created if no other has already been specified by the user in `Cluster::stores`.
`Cluster::default_persistent_backend`: `Broker::BackendType` `&redef`	The type of persistent data store backend that will be used for all data stores if no other has already been specified by the user in `Cluster::stores`.
`Cluster::default_store_dir`: `string` `&redef`	Setting a default dir will, for persistent backends that have not been given an explicit file path via `Cluster::stores`, automatically create a path within this dir that is based on the name of the data store.
`Cluster::enable_round_robin_logging`: `bool` `&redef`	Whether to distribute log messages among available logging nodes.
`Cluster::logger_topic`: `string` `&redef`	The topic name used for exchanging messages that are relevant to logger nodes in a cluster.
`Cluster::manager_is_logger`: `bool` `&redef`	Indicates whether or not the manager will act as the logger and receive logs.
`Cluster::manager_topic`: `string` `&redef`	The topic name used for exchanging messages that are relevant to manager nodes in a cluster.
`Cluster::node`: `string` `&redef`	This is usually supplied on the command line for each instance of the cluster that is started up.
`Cluster::node_topic_prefix`: `string` `&redef`	The topic prefix used for exchanging messages that are relevant to a named node in a cluster.
`Cluster::nodeid_topic_prefix`: `string` `&redef`	The topic prefix used for exchanging messages that are relevant to a unique node in a cluster.
`Cluster::nodes`: `table` `&redef`	The cluster layout definition.
`Cluster::proxy_topic`: `string` `&redef`	The topic name used for exchanging messages that are relevant to proxy nodes in a cluster.
`Cluster::retry_interval`: `interval` `&redef`	Interval for retrying failed connections between cluster nodes.
`Cluster::worker_topic`: `string` `&redef`	The topic name used for exchanging messages that are relevant to worker nodes in a cluster.

Constants

Cluster::broadcast_topics: set

A set of topic names to be used for broadcasting messages that are relevant to all nodes in a cluster.

State Variables

Cluster::stores: table &default = … &redef

A table of cluster-enabled data stores that have been created, indexed by their name.

Types

`Cluster::Info`: `record` `&log`	The record type which contains the column fields of the cluster log.
`Cluster::StoreInfo`: `record`	Information regarding a cluster-enabled data store.
`Cluster::BackendTag`: `enum`
`Cluster::EventSerializerTag`: `enum`
`Cluster::LogSerializerTag`: `enum`

Redefinitions

Log::ID: enum

The cluster logging stream identifier.

Cluster::LOG

Events

`Cluster::hello`: `event`	When using broker-enabled cluster framework, nodes broadcast this event to exchange their user-defined name along with a string that uniquely identifies it for the duration of its lifetime.
`Cluster::node_down`: `event`	When using broker-enabled cluster framework, this event will be emitted locally whenever a connected cluster node becomes disconnected.
`Cluster::node_up`: `event`	When using broker-enabled cluster framework, this event will be emitted locally whenever a cluster node connects or reconnects.

Hooks

`Cluster::connect_node_hook`: `hook`	This hook is called when the local node connects to other nodes based on the given cluster layout.
`Cluster::log_policy`: `Log::PolicyHook`	A default logging policy hook for the stream.

Functions

`Cluster::create_store`: `function` `&deprecated` = …	Sets up a cluster-enabled data store.
`Cluster::get_active_node_count`: `function`	Returns the number of nodes per type, the calling node is currently connected to.
`Cluster::get_node_count`: `function`	Returns the number of nodes defined in the cluster layout for a given node type.
`Cluster::init`: `function`	Initialize the cluster backend.
`Cluster::is_enabled`: `function`	This function can be called at any time to determine if the cluster framework is being enabled for this run.
`Cluster::listen_websocket`: `function`	Start listening on a WebSocket address.
`Cluster::local_node_metrics_port`: `function`	This function can be called at any time to determine the configured metrics port for Prometheus being used by current Zeek instance.
`Cluster::local_node_type`: `function`	This function can be called at any time to determine what type of cluster node the current Zeek instance is going to be acting as.
`Cluster::log`: `function`	Write a message to the cluster logging stream.
`Cluster::node_id`: `function` `&redef`	Function returning this node’s identifier.
`Cluster::node_topic`: `function` `&redef`	Retrieve the topic associated with a specific node in the cluster.
`Cluster::nodeid_to_node`: `function`	Retrieve the cluster-level naming of a node based on its node ID, a backend-specific identifier.
`Cluster::nodeid_topic`: `function` `&redef`	Retrieve the topic associated with a specific node in the cluster.

Detailed Interface

Redefinable Options

Cluster::default_backend 

Type:: Broker::BackendType
Attributes:: &redef
Default:: Broker::MEMORY

The type of data store backend that will be used for all data stores if no other has already been specified by the user in Cluster::stores.

Cluster::default_master_node 

Type:: string
Attributes:: &redef
Default:: ""

Name of the node on which master data stores will be created if no other has already been specified by the user in Cluster::stores. An empty value means “use whatever name corresponds to the manager node”.

Cluster::default_persistent_backend 

Type:: Broker::BackendType
Attributes:: &redef
Default:: Broker::SQLITE

The type of persistent data store backend that will be used for all data stores if no other has already been specified by the user in Cluster::stores. This will be used when script authors call Cluster::create_store with the persistent argument set true.

Cluster::default_store_dir 

Type:: string
Attributes:: &redef
Default:: ""

Setting a default dir will, for persistent backends that have not been given an explicit file path via Cluster::stores, automatically create a path within this dir that is based on the name of the data store.

Cluster::enable_round_robin_logging 

Type:: bool
Attributes:: &redef
Default:: T

Whether to distribute log messages among available logging nodes.

Cluster::logger_topic 

Type:

string

Attributes:

&redef

Default:

"zeek/cluster/logger"

Redefinition:

from policy/frameworks/cluster/backend/zeromq/main.zeek

=:

``zeek.cluster.logger``

The topic name used for exchanging messages that are relevant to logger nodes in a cluster. Used with broker-enabled cluster communication.

Cluster::manager_is_logger 

Type:: bool
Attributes:: &redef
Default:: T

Indicates whether or not the manager will act as the logger and receive logs. This value should be set in the cluster-layout.zeek script (the value should be true only if no logger is specified in Cluster::nodes). Note that ZeekControl handles this automatically.

Cluster::manager_topic 

Type:

string

Attributes:

&redef

Default:

"zeek/cluster/manager"

Redefinition:

from policy/frameworks/cluster/backend/zeromq/main.zeek

=:

``zeek.cluster.manager``

The topic name used for exchanging messages that are relevant to manager nodes in a cluster. Used with broker-enabled cluster communication.

Cluster::node 

Type:: string
Attributes:: &redef
Default:: ""

This is usually supplied on the command line for each instance of the cluster that is started up.

Cluster::node_topic_prefix 

Type:: string
Attributes:: &redef
Default:: "zeek/cluster/node/"

The topic prefix used for exchanging messages that are relevant to a named node in a cluster. Used with broker-enabled cluster communication.

Cluster::nodeid_topic_prefix 

Type:: string
Attributes:: &redef
Default:: "zeek/cluster/nodeid/"

The topic prefix used for exchanging messages that are relevant to a unique node in a cluster. Used with broker-enabled cluster communication.

Cluster::nodes 

Type:: table [string] of Cluster::Node
Attributes:: &redef
Default:: {}

The cluster layout definition. This should be placed into a filter named cluster-layout.zeek somewhere in the ZEEKPATH. It will be automatically loaded if the CLUSTER_NODE environment variable is set. Note that ZeekControl handles all of this automatically. The table is typically indexed by node names/labels (e.g. “manager” or “worker-1”).

Cluster::proxy_topic 

Type:

string

Attributes:

&redef

Default:

"zeek/cluster/proxy"

Redefinition:

from policy/frameworks/cluster/backend/zeromq/main.zeek

=:

``zeek.cluster.proxy``

The topic name used for exchanging messages that are relevant to proxy nodes in a cluster. Used with broker-enabled cluster communication.

Cluster::retry_interval 

Type:: interval
Attributes:: &redef
Default:: 1.0 sec

Interval for retrying failed connections between cluster nodes. If set, the ZEEK_DEFAULT_CONNECT_RETRY (given in number of seconds) environment variable overrides this option.

Cluster::worker_topic 

Type:

string

Attributes:

&redef

Default:

"zeek/cluster/worker"

Redefinition:

from policy/frameworks/cluster/backend/zeromq/main.zeek

=:

``zeek.cluster.worker``

The topic name used for exchanging messages that are relevant to worker nodes in a cluster. Used with broker-enabled cluster communication.

Constants

Cluster::broadcast_topics 

Type:

set [string]

Default:

{
   "zeek/cluster/manager",
   "zeek/cluster/logger",
   "zeek/cluster/proxy",
   "zeek/cluster/worker"
}

A set of topic names to be used for broadcasting messages that are relevant to all nodes in a cluster. Currently, there is not a common topic to broadcast to, because enabling implicit Broker forwarding would cause a routing loop for this topic.

State Variables

Cluster::stores 

Type:: table [string] of Cluster::StoreInfo
Attributes:: &default = [name=<uninitialized>, store=<uninitialized>, master_node=, master=F, backend=Broker::MEMORY, options=[sqlite=[path=, synchronous=<uninitialized>, journal_mode=<uninitialized>, failure_mode=Broker::SQLITE_FAILURE_MODE_FAIL, integrity_check=F]], clone_resync_interval=10.0 secs, clone_stale_interval=5.0 mins, clone_mutation_buffer_interval=2.0 mins] &redef
Default:: {}

A table of cluster-enabled data stores that have been created, indexed by their name. This table will be populated automatically by Cluster::create_store, but if you need to customize the options related to a particular data store, you may redef this table. Calls to Cluster::create_store will first check the table for an entry of the same name and, if found, will use the predefined options there when setting up the store.

Types

Cluster::Info 

Type:

record

Fields:

ts: time &log: The time at which a cluster message was generated.

node: string &log: The name of the node that is creating the log record.

message: string &log: A message indicating information about the cluster’s operation.

Attributes:: &log

The record type which contains the column fields of the cluster log.

Cluster::StoreInfo 

Type:

record

Fields:

name: string &optional: The name of the data store.

store: opaque of Broker::Store &optional: The store handle.

master_node: string &default = Cluster::default_master_node &optional: The name of the cluster node on which the master version of the data store resides.

master: bool &default = F &optional: Whether the data store is the master version or a clone.

backend: Broker::BackendType &default = Cluster::default_backend &optional: The type of backend used for storing data.

options: Broker::BackendOptions &default = ... &optional: Parameters used for configuring the backend.

clone_resync_interval: interval &default = Broker::default_clone_resync_interval &optional: A resync/reconnect interval to pass through to Broker::create_clone.

clone_stale_interval: interval &default = Broker::default_clone_stale_interval &optional: A staleness duration to pass through to Broker::create_clone.

clone_mutation_buffer_interval: interval &default = Broker::default_clone_mutation_buffer_interval &optional: A mutation buffer interval to pass through to Broker::create_clone.

Information regarding a cluster-enabled data store.

Cluster::BackendTag

Type:

enum

Cluster::CLUSTER_BACKEND_BROKER

Cluster::CLUSTER_BACKEND_BROKER_WEBSOCKET_SHIM

Cluster::CLUSTER_BACKEND_NONE

Cluster::CLUSTER_BACKEND_ZEROMQ

Cluster::EventSerializerTag

Type:

enum

Cluster::EVENT_SERIALIZER_BROKER_BIN_V1

Cluster::EVENT_SERIALIZER_BROKER_JSON_V1

Cluster::LogSerializerTag

Type:

enum

Cluster::LOG_SERIALIZER_ZEEK_BIN_V1

Events

Cluster::hello 

Type:: event (name: string, id: string)

When using broker-enabled cluster framework, nodes broadcast this event to exchange their user-defined name along with a string that uniquely identifies it for the duration of its lifetime. This string may change if the node dies and has to reconnect later.

Cluster::node_down 

Type:: event (name: string, id: string)

When using broker-enabled cluster framework, this event will be emitted locally whenever a connected cluster node becomes disconnected.

Cluster::node_up 

Type:: event (name: string, id: string)

When using broker-enabled cluster framework, this event will be emitted locally whenever a cluster node connects or reconnects.

Hooks

Cluster::connect_node_hook 

Type:: hook (connectee: Cluster::NamedNode) : bool

This hook is called when the local node connects to other nodes based on the given cluster layout. Breaking from the hook will prevent connection establishment.

This hook only applies to the Broker cluster backend.

Parameters:: connectee – The node to connect to.

Cluster::log_policy 

Type:: Log::PolicyHook

A default logging policy hook for the stream.

Functions

Cluster::create_store 

Type:: function (name: string, persistent: bool &default = F &optional) : Cluster::StoreInfo
Attributes:: &deprecated = “Remove in v9.1. Cluster::create_store() uses Broker stores which are deprecated. To distribute state across cluster nodes, use the new &publish_on_change attribute for global sets/tables, or leverage explicit remote events with Cluster::publish(). For state persistence, use the storage framework.”

Sets up a cluster-enabled data store. They will also still properly function for uses that are not operating a cluster.

Parameters:

name – the name of the data store to create.
persistent – whether the data store must be persistent.

Returns:

the store’s information. For master stores, the store will be ready to use immediately. For clones, the store field will not be set until the node containing the master store has connected.

Cluster::get_active_node_count 

Type:: function (node_type: Cluster::NodeType) : count

Returns the number of nodes per type, the calling node is currently connected to. This is primarily intended for use by the manager to find out how many nodes should be responding to requests.

Cluster::get_node_count 

Type:: function (node_type: Cluster::NodeType) : count

Returns the number of nodes defined in the cluster layout for a given node type.

Cluster::init 

Type:: function () : bool

Initialize the cluster backend.

Cluster backends usually invoke this from a zeek_init handler.

Returns:: T on success, else F.

Cluster::is_enabled 

Type:: function () : bool

This function can be called at any time to determine if the cluster framework is being enabled for this run.

Returns:: True if Cluster::node has been set.

Cluster::listen_websocket 

Type:: function (options: Cluster::WebSocketServerOptions) : bool

Start listening on a WebSocket address.

Parameters:: options – The server Cluster::WebSocketServerOptions to use.
Returns:: T on success, else F.

Cluster::local_node_metrics_port 

Type:: function () : port

This function can be called at any time to determine the configured metrics port for Prometheus being used by current Zeek instance. If Cluster::is_enabled returns false or the node isn’t found, 0/unknown is returned.

Returns:: The metrics port used by the calling node.

Cluster::local_node_type 

Type:: function () : Cluster::NodeType

This function can be called at any time to determine what type of cluster node the current Zeek instance is going to be acting as. If Cluster::is_enabled returns false, then Cluster::NONE is returned.

Returns:: The Cluster::NodeType the calling node acts as.

Cluster::log 

Type:: function (msg: string) : void

Write a message to the cluster logging stream.

Cluster::node_id 

Type:: function () : string
Attributes:: &redef

Function returning this node’s identifier.

By default this is Broker::node_id, but can be redefined by other cluster backends. This identifier should be a short lived identifier that resets when a node is restarted.

Cluster::node_topic 

Type:: function (name: string) : string
Attributes:: &redef

Retrieve the topic associated with a specific node in the cluster.

Parameters:: name – the name of the cluster node (e.g. “manager”).
Returns:: a topic string that may used to send a message exclusively to a given cluster node.

Cluster::nodeid_to_node 

Type:: function (id: string) : Cluster::NamedNode

Retrieve the cluster-level naming of a node based on its node ID, a backend-specific identifier.

Parameters:: id – the node ID of a peer.
Returns:: the Cluster::NamedNode for the requested node, if known, otherwise a “null” instance with an empty name field.

Cluster::nodeid_topic 

Type:: function (id: string) : string
Attributes:: &redef

Retrieve the topic associated with a specific node in the cluster.

Parameters:: id – the id of the cluster node (from Broker::EndpointInfo or Broker::node_id.
Returns:: a topic string that may used to send a message exclusively to a given cluster node.