base/frameworks/cluster/main.zeek¶
- Cluster¶
A framework for establishing and controlling a cluster of Zeek instances.
In order to use the cluster framework, a script named
cluster-layout.zeek
must exist somewhere in Zeek’s script search path
which has a cluster definition of the Cluster::nodes
variable.
The CLUSTER_NODE
environment variable or Cluster::node
must also be sent and the cluster framework loaded as a package like
@load base/frameworks/cluster
.
Warning
The file cluster-layout.zeek
should only contain the definition
of Cluster::nodes
. Specifically, avoid loading other Zeek
scripts or using redef
for anything but Cluster::nodes
.
Due to cluster-layout.zeek
being loaded very early, it is easy to
introduce circular loading issues.
- Namespace
Cluster
- Imports
Summary¶
Redefinable Options¶
The type of data store backend that will be used for all data stores if
no other has already been specified by the user in |
|
Name of the node on which master data stores will be created if no other
has already been specified by the user in |
|
|
The type of persistent data store backend that will be used for all data
stores if no other has already been specified by the user in
|
Setting a default dir will, for persistent backends that have not
been given an explicit file path via |
|
Whether to distribute log messages among available logging nodes. |
|
The topic name used for exchanging messages that are relevant to logger nodes in a cluster. |
|
Indicates whether or not the manager will act as the logger and receive logs. |
|
The topic name used for exchanging messages that are relevant to manager nodes in a cluster. |
|
This is usually supplied on the command line for each instance of the cluster that is started up. |
|
The topic prefix used for exchanging messages that are relevant to a named node in a cluster. |
|
The topic prefix used for exchanging messages that are relevant to a unique node in a cluster. |
|
The cluster layout definition. |
|
The topic name used for exchanging messages that are relevant to proxy nodes in a cluster. |
|
Interval for retrying failed connections between cluster nodes. |
|
The topic name used for exchanging messages that are relevant to time machine nodes in a cluster. |
|
The topic name used for exchanging messages that are relevant to worker nodes in a cluster. |
Constants¶
A set of topic names to be used for broadcasting messages that are relevant to all nodes in a cluster. |
State Variables¶
A table of cluster-enabled data stores that have been created, indexed by their name. |
|
This gives the value for the number of workers currently connected to, and it’s maintained internally by the cluster framework. |
Types¶
The record type which contains the column fields of the cluster log. |
|
Record to represent a cluster node including its name. |
|
Record type to indicate a node in a cluster. |
|
Types of nodes that are allowed to participate in the cluster configuration. |
|
Information regarding a cluster-enabled data store. |
Redefinitions¶
The cluster logging stream identifier. |
Events¶
When using broker-enabled cluster framework, nodes broadcast this event to exchange their user-defined name along with a string that uniquely identifies it for the duration of its lifetime. |
|
When using broker-enabled cluster framework, this event will be emitted locally whenever a connected cluster node becomes disconnected. |
|
When using broker-enabled cluster framework, this event will be emitted locally whenever a cluster node connects or reconnects. |
Hooks¶
A default logging policy hook for the stream. |
Functions¶
Sets up a cluster-enabled data store. |
|
Returns the number of nodes per type, the calling node is currently connected to. |
|
Returns the number of nodes defined in the cluster layout for a given node type. |
|
This function can be called at any time to determine if the cluster framework is being enabled for this run. |
|
This function can be called at any time to determine what type of cluster node the current Zeek instance is going to be acting as. |
|
Write a message to the cluster logging stream. |
|
Retrieve the topic associated with a specific node in the cluster. |
|
Retrieve the topic associated with a specific node in the cluster. |
Detailed Interface¶
Redefinable Options¶
- Cluster::default_backend¶
- Type
- Attributes
- Default
Broker::MEMORY
The type of data store backend that will be used for all data stores if no other has already been specified by the user in
Cluster::stores
.
- Cluster::default_master_node¶
-
Name of the node on which master data stores will be created if no other has already been specified by the user in
Cluster::stores
. An empty value means “use whatever name corresponds to the manager node”.
- Cluster::default_persistent_backend¶
- Type
- Attributes
- Default
Broker::SQLITE
The type of persistent data store backend that will be used for all data stores if no other has already been specified by the user in
Cluster::stores
. This will be used when script authors callCluster::create_store
with the persistent argument set true.
- Cluster::default_store_dir¶
-
Setting a default dir will, for persistent backends that have not been given an explicit file path via
Cluster::stores
, automatically create a path within this dir that is based on the name of the data store.
- Cluster::enable_round_robin_logging¶
-
Whether to distribute log messages among available logging nodes.
- Cluster::logger_topic¶
-
The topic name used for exchanging messages that are relevant to logger nodes in a cluster. Used with broker-enabled cluster communication.
- Cluster::manager_is_logger¶
-
Indicates whether or not the manager will act as the logger and receive logs. This value should be set in the cluster-layout.zeek script (the value should be true only if no logger is specified in Cluster::nodes). Note that ZeekControl handles this automatically.
- Cluster::manager_topic¶
-
The topic name used for exchanging messages that are relevant to manager nodes in a cluster. Used with broker-enabled cluster communication.
- Cluster::node¶
-
This is usually supplied on the command line for each instance of the cluster that is started up.
- Cluster::node_topic_prefix¶
-
The topic prefix used for exchanging messages that are relevant to a named node in a cluster. Used with broker-enabled cluster communication.
- Cluster::nodeid_topic_prefix¶
-
The topic prefix used for exchanging messages that are relevant to a unique node in a cluster. Used with broker-enabled cluster communication.
- Cluster::nodes¶
- Type
table
[string
] ofCluster::Node
- Attributes
- Default
{}
The cluster layout definition. This should be placed into a filter named cluster-layout.zeek somewhere in the ZEEKPATH. It will be automatically loaded if the CLUSTER_NODE environment variable is set. Note that ZeekControl handles all of this automatically. The table is typically indexed by node names/labels (e.g. “manager” or “worker-1”).
- Cluster::proxy_topic¶
-
The topic name used for exchanging messages that are relevant to proxy nodes in a cluster. Used with broker-enabled cluster communication.
- Cluster::retry_interval¶
-
Interval for retrying failed connections between cluster nodes. If set, the ZEEK_DEFAULT_CONNECT_RETRY (given in number of seconds) environment variable overrides this option.
- Cluster::time_machine_topic¶
-
The topic name used for exchanging messages that are relevant to time machine nodes in a cluster. Used with broker-enabled cluster communication.
- Cluster::worker_topic¶
-
The topic name used for exchanging messages that are relevant to worker nodes in a cluster. Used with broker-enabled cluster communication.
Constants¶
- Cluster::broadcast_topics¶
- Type
- Default
{ "zeek/cluster/manager", "zeek/cluster/logger", "zeek/cluster/proxy", "zeek/cluster/worker", "zeek/cluster/time_machine" }
A set of topic names to be used for broadcasting messages that are relevant to all nodes in a cluster. Currently, there is not a common topic to broadcast to, because enabling implicit Broker forwarding would cause a routing loop for this topic.
State Variables¶
- Cluster::stores¶
- Type
- Attributes
&default
= [name=<uninitialized>, store=<uninitialized>, master_node=, master=F, backend=Broker::MEMORY, options=[sqlite=[path=, synchronous=<uninitialized>, journal_mode=<uninitialized>, failure_mode=Broker::SQLITE_FAILURE_MODE_FAIL, integrity_check=F]], clone_resync_interval=10.0 secs, clone_stale_interval=5.0 mins, clone_mutation_buffer_interval=2.0 mins]&redef
- Default
{}
A table of cluster-enabled data stores that have been created, indexed by their name. This table will be populated automatically by
Cluster::create_store
, but if you need to customize the options related to a particular data store, you may redef this table. Calls toCluster::create_store
will first check the table for an entry of the same name and, if found, will use the predefined options there when setting up the store.
- Cluster::worker_count¶
- Type
- Attributes
&deprecated
= “Remove in v6.1. Active worker count can be obtained via get_active_node_count(Cluster::WORKER)”- Default
0
This gives the value for the number of workers currently connected to, and it’s maintained internally by the cluster framework. It’s primarily intended for use by managers to find out how many workers should be responding to requests.
Types¶
- Cluster::Info¶
- Type
- Attributes
The record type which contains the column fields of the cluster log.
- Cluster::NamedNode¶
- Type
-
name:
string
node:
Cluster::Node
Record to represent a cluster node including its name.
- Cluster::Node¶
- Type
-
- node_type:
Cluster::NodeType
Identifies the type of cluster node in this node’s configuration.
- ip:
addr
The IP address of the cluster node.
- zone_id:
string
&default
=""
&optional
If the ip field is a non-global IPv6 address, this field can specify a particular RFC 4007
zone_id
.- p:
port
&default
=0/unknown
&optional
The port that this node will listen on for peer connections. A value of
0/unknown
means the node is not pre-configured to listen.- interface:
string
&optional
Identifier for the interface a worker is sniffing.
- manager:
string
&optional
Name of the manager node this node uses. For workers and proxies.
- time_machine:
string
&optional
Name of a time machine node with which this node connects.
- id:
string
&optional
A unique identifier assigned to the node by the broker framework. This field is only set while a node is connected.
- lb_filter:
string
&optional
(present if policy/misc/load-balancing.zeek is loaded)
A BPF filter for load balancing traffic sniffed on a single interface across a number of processes. In normal uses, this will be assigned dynamically by the manager and installed by the workers.
- node_type:
Record type to indicate a node in a cluster.
- Cluster::NodeType¶
- Type
-
- Cluster::NONE¶
A dummy node type indicating the local node is not operating within a cluster.
- Cluster::CONTROL¶
A node type which is allowed to view/manipulate the configuration of other nodes in the cluster.
- Cluster::LOGGER¶
A node type responsible for log management.
- Cluster::MANAGER¶
A node type responsible for policy management.
- Cluster::PROXY¶
A node type for relaying worker node communication and synchronizing worker node state.
- Cluster::WORKER¶
The node type doing all the actual traffic analysis.
- Cluster::TIME_MACHINE¶
A node acting as a traffic recorder using the Time Machine software.
Types of nodes that are allowed to participate in the cluster configuration.
- Cluster::StoreInfo¶
- Type
-
- name:
string
&optional
The name of the data store.
- store:
opaque
of Broker::Store&optional
The store handle.
- master_node:
string
&default
=Cluster::default_master_node
&optional
The name of the cluster node on which the master version of the data store resides.
- master:
bool
&default
=F
&optional
Whether the data store is the master version or a clone.
- backend:
Broker::BackendType
&default
=Cluster::default_backend
&optional
The type of backend used for storing data.
- options:
Broker::BackendOptions
&default
= [sqlite=[path=, synchronous=<uninitialized>, journal_mode=<uninitialized>, failure_mode=Broker::SQLITE_FAILURE_MODE_FAIL, integrity_check=F]]&optional
Parameters used for configuring the backend.
- clone_resync_interval:
interval
&default
=Broker::default_clone_resync_interval
&optional
A resync/reconnect interval to pass through to
Broker::create_clone
.- clone_stale_interval:
interval
&default
=Broker::default_clone_stale_interval
&optional
A staleness duration to pass through to
Broker::create_clone
.- clone_mutation_buffer_interval:
interval
&default
=Broker::default_clone_mutation_buffer_interval
&optional
A mutation buffer interval to pass through to
Broker::create_clone
.
- name:
Information regarding a cluster-enabled data store.
Events¶
- Cluster::hello¶
-
When using broker-enabled cluster framework, nodes broadcast this event to exchange their user-defined name along with a string that uniquely identifies it for the duration of its lifetime. This string may change if the node dies and has to reconnect later.
- Cluster::node_down¶
-
When using broker-enabled cluster framework, this event will be emitted locally whenever a connected cluster node becomes disconnected.
- Cluster::node_up¶
-
When using broker-enabled cluster framework, this event will be emitted locally whenever a cluster node connects or reconnects.
Hooks¶
- Cluster::log_policy¶
- Type
A default logging policy hook for the stream.
Functions¶
- Cluster::create_store¶
-
Sets up a cluster-enabled data store. They will also still properly function for uses that are not operating a cluster.
- Parameters
name – the name of the data store to create.
persistent – whether the data store must be persistent.
- Returns
the store’s information. For master stores, the store will be ready to use immediately. For clones, the store field will not be set until the node containing the master store has connected.
- Cluster::get_active_node_count¶
- Type
function
(node_type:Cluster::NodeType
) :count
Returns the number of nodes per type, the calling node is currently connected to. This is primarily intended for use by the manager to find out how many nodes should be responding to requests.
- Cluster::get_node_count¶
- Type
function
(node_type:Cluster::NodeType
) :count
Returns the number of nodes defined in the cluster layout for a given node type.
- Cluster::is_enabled¶
-
This function can be called at any time to determine if the cluster framework is being enabled for this run.
- Returns
True if
Cluster::node
has been set.
- Cluster::local_node_type¶
- Type
This function can be called at any time to determine what type of cluster node the current Zeek instance is going to be acting as. If
Cluster::is_enabled
returns false, thenCluster::NONE
is returned.- Returns
The
Cluster::NodeType
the calling node acts as.
- Cluster::log¶
-
Write a message to the cluster logging stream.
- Cluster::node_topic¶
-
Retrieve the topic associated with a specific node in the cluster.
- Parameters
name – the name of the cluster node (e.g. “manager”).
- Returns
a topic string that may used to send a message exclusively to a given cluster node.
- Cluster::nodeid_topic¶
-
Retrieve the topic associated with a specific node in the cluster.
- Parameters
id – the id of the cluster node (from
Broker::EndpointInfo
orBroker::node_id
.- Returns
a topic string that may used to send a message exclusively to a given cluster node.