base/frameworks/broker/main.zeek
- Broker
The Broker-based communication API and its various options.
- Namespace
Broker
- Imports
Summary
Runtime Options
Whether calling |
Redefinable Options
Frequency of work-stealing polling attempts for Broker/CAF threads in “aggressive” mode. |
|
Number of work-stealing polling attempts for Broker/CAF threads in “aggressive” mode. |
|
The number of buffered messages at the Broker/CAF layer after which a subscriber considers themselves congested (i.e. |
|
Default interval to retry connecting to a peer if it cannot be made to work initially, or if it ever becomes disconnected. |
|
Default address on which to listen. |
|
Default address on which to listen for WebSocket connections. |
|
Default interval to retry listening on a port if it’s currently in use already. |
|
The default topic prefix where logs will be published. |
|
Default port for native Broker communication. |
|
Default port for Broker WebSocket communication. |
|
If true, do not use SSL for network connections. |
|
Forward all received messages to subscribing peers. |
|
Max time to buffer log messages before sending the current set out as a batch. |
|
The max number of log entries per log stream to batch together when sending log messages to a remote logger. |
|
Max number of threads to use for Broker/CAF functionality. |
|
Frequency of work-stealing polling attempts for Broker/CAF threads in “moderate” mode. |
|
Number of work-stealing polling attempts for Broker/CAF threads in “moderate” mode. |
|
Interval of time for under-utilized Broker/CAF threads to sleep when in “moderate” mode. |
|
Frequency of work-stealing polling attempts for Broker/CAF threads in “relaxed” mode. |
|
Interval of time for under-utilized Broker/CAF threads to sleep when in “relaxed” mode. |
|
The CAF scheduling policy to use. |
|
Path to a file containing concatenated trusted certificates in PEM format. |
|
Path to an OpenSSL-style directory of trusted certificates. |
|
Path to a file containing a X.509 certificate for this node in PEM format. |
|
Path to the file containing the private key for this node’s certificate. |
|
Passphrase to decrypt the private key specified by
|
Types
Opaque communication data. |
|
Opaque communication data sequence. |
|
Enumerates the possible error types. |
|
Opaque event communication data. |
|
The possible states of a peer endpoint. |
|
Opaque communication data used as a convenient way to wrap key-value pairs that comprise table entries. |
Functions
Automatically send an event to any interested peers whenever it is locally dispatched. |
|
Stop automatically sending an event to peers upon local dispatch. |
|
The default implementation for |
|
Sends all pending log messages to remote peers. |
|
Register a topic prefix subscription for events that should only be forwarded to any subscribing peers and not raise any event handlers on the receiving/forwarding node. |
|
Listen for remote connections using the native Broker protocol. |
|
Listen for remote connections using WebSocket. |
|
A function that will be called for each log entry to determine what broker topic string will be used for sending it to peers. |
|
Get a unique identifier for the local broker endpoint. |
|
Initiate a remote connection. |
|
Get a list of all peer connections. |
|
Publishes the value of an identifier to a given topic. |
|
Register interest in all peer event messages that use a certain topic prefix. |
|
Remove a remote connection. |
|
Unregister interest in all peer event messages that use a topic prefix. |
Detailed Interface
Runtime Options
- Broker::peer_counts_as_iosource
-
Whether calling
Broker::peer
will register the Broker system as an I/O source that will block the process from shutting down. For example, set this to false when you are reading pcaps, but also want to initiate a Broker peering and still shutdown after done reading the pcap.
Redefinable Options
- Broker::aggressive_interval
-
Frequency of work-stealing polling attempts for Broker/CAF threads in “aggressive” mode. Only used for the “stealing” scheduler policy.
- Broker::aggressive_polls
-
Number of work-stealing polling attempts for Broker/CAF threads in “aggressive” mode. Only used for the “stealing” scheduler policy.
- Broker::congestion_queue_size
-
The number of buffered messages at the Broker/CAF layer after which a subscriber considers themselves congested (i.e. tune the congestion control mechanisms).
- Broker::default_connect_retry
-
Default interval to retry connecting to a peer if it cannot be made to work initially, or if it ever becomes disconnected. Use of the ZEEK_DEFAULT_CONNECT_RETRY environment variable (set as number of seconds) will override this option and also any values given to
Broker::peer
.
- Broker::default_listen_address
- Type
- Attributes
- Default
""
- Redefinition
from policy/frameworks/management/agent/boot.zeek
=
:127.0.0.1
Default address on which to listen.
See also:
Broker::listen
- Broker::default_listen_address_websocket
-
Default address on which to listen for WebSocket connections.
See also:
Broker::listen_websocket
- Broker::default_listen_retry
-
Default interval to retry listening on a port if it’s currently in use already. Use of the ZEEK_DEFAULT_LISTEN_RETRY environment variable (set as a number of seconds) will override this option and also any values given to
Broker::listen
.
- Broker::default_log_topic_prefix
-
The default topic prefix where logs will be published. The log’s stream id is appended when writing to a particular stream.
- Broker::default_port
-
Default port for native Broker communication. Where not specified otherwise, this is the port to connect to and listen on.
- Broker::default_port_websocket
-
Default port for Broker WebSocket communication. Where not specified otherwise, this is the port to connect to and listen on for WebSocket connections.
See the Broker documentation for a specification of the message format over WebSocket connections.
- Broker::disable_ssl
-
If true, do not use SSL for network connections. By default, SSL will even be used if no certificates / CAs have been configured. In that case (which is the default) the communication will be encrypted, but not authenticated.
- Broker::forward_messages
-
Forward all received messages to subscribing peers.
- Broker::log_batch_interval
-
Max time to buffer log messages before sending the current set out as a batch.
- Broker::log_batch_size
-
The max number of log entries per log stream to batch together when sending log messages to a remote logger.
- Broker::max_threads
-
Max number of threads to use for Broker/CAF functionality. The ZEEK_BROKER_MAX_THREADS environment variable overrides this setting.
- Broker::moderate_interval
-
Frequency of work-stealing polling attempts for Broker/CAF threads in “moderate” mode. Only used for the “stealing” scheduler policy.
- Broker::moderate_polls
-
Number of work-stealing polling attempts for Broker/CAF threads in “moderate” mode. Only used for the “stealing” scheduler policy.
- Broker::moderate_sleep
-
Interval of time for under-utilized Broker/CAF threads to sleep when in “moderate” mode. Only used for the “stealing” scheduler policy.
- Broker::relaxed_interval
-
Frequency of work-stealing polling attempts for Broker/CAF threads in “relaxed” mode. Only used for the “stealing” scheduler policy.
- Broker::relaxed_sleep
-
Interval of time for under-utilized Broker/CAF threads to sleep when in “relaxed” mode. Only used for the “stealing” scheduler policy.
- Broker::scheduler_policy
-
The CAF scheduling policy to use. Available options are “sharing” and “stealing”. The “sharing” policy uses a single, global work queue along with mutex and condition variable used for accessing it, which may be better for cases that don’t require much concurrency or need lower power consumption. The “stealing” policy uses multiple work queues protected by spinlocks, which may be better for use-cases that have more concurrency needs. E.g. may be worth testing the “stealing” policy along with dedicating more threads if a lot of data store processing is required.
- Broker::ssl_cafile
-
Path to a file containing concatenated trusted certificates in PEM format. If set, Zeek will require valid certificates for all peers.
- Broker::ssl_capath
-
Path to an OpenSSL-style directory of trusted certificates. If set, Zeek will require valid certificates for all peers.
- Broker::ssl_certificate
-
Path to a file containing a X.509 certificate for this node in PEM format. If set, Zeek will require valid certificates for all peers.
- Broker::ssl_keyfile
-
Path to the file containing the private key for this node’s certificate. If set, Zeek will require valid certificates for all peers.
- Broker::ssl_passphrase
-
Passphrase to decrypt the private key specified by
Broker::ssl_keyfile
. If set, Zeek will require valid certificates for all peers.
Types
- Broker::Data
-
Opaque communication data.
- Broker::DataVector
- Type
Opaque communication data sequence.
- Broker::EndpointInfo
- Type
-
- id:
string
A unique identifier of the node.
- network:
Broker::NetworkInfo
&optional
Network-level information.
- id:
- Broker::ErrorCode
- Type
-
- Broker::NO_ERROR
(present if base/bif/comm.bif.zeek is loaded)
- Broker::UNSPECIFIED
The unspecified default error code.
- Broker::PEER_INCOMPATIBLE
Version incompatibility.
- Broker::PEER_INVALID
Referenced peer does not exist.
- Broker::PEER_UNAVAILABLE
Remote peer not listening.
- Broker::PEER_DISCONNECT_DURING_HANDSHAKE
(present if base/bif/comm.bif.zeek is loaded)
- Broker::PEER_TIMEOUT
A peering request timed out.
- Broker::MASTER_EXISTS
Master with given name already exists.
- Broker::NO_SUCH_MASTER
Master with given name does not exist.
- Broker::NO_SUCH_KEY
The given data store key does not exist.
- Broker::REQUEST_TIMEOUT
The store operation timed out.
- Broker::TYPE_CLASH
The operation expected a different type than provided.
- Broker::INVALID_DATA
The data value cannot be used to carry out the desired operation.
- Broker::BACKEND_FAILURE
The storage backend failed to execute the operation.
- Broker::STALE_DATA
The storage backend failed to execute the operation.
- Broker::CANNOT_OPEN_FILE
(present if base/bif/comm.bif.zeek is loaded)
- Broker::CANNOT_WRITE_FILE
(present if base/bif/comm.bif.zeek is loaded)
- Broker::INVALID_TOPIC_KEY
(present if base/bif/comm.bif.zeek is loaded)
- Broker::END_OF_FILE
(present if base/bif/comm.bif.zeek is loaded)
- Broker::INVALID_TAG
(present if base/bif/comm.bif.zeek is loaded)
- Broker::INVALID_STATUS
(present if base/bif/comm.bif.zeek is loaded)
- Broker::CAF_ERROR
Catch-all for a CAF-level problem.
Enumerates the possible error types.
- Broker::Event
- Type
-
- name:
string
&optional
The name of the event. Not set if invalid event or arguments.
- args:
Broker::DataVector
The arguments to the event.
- name:
Opaque event communication data.
- Broker::PeerInfo
- Type
-
peer:
Broker::EndpointInfo
status:
Broker::PeerStatus
- Broker::PeerInfos
- Type
- Broker::PeerStatus
- Type
-
- Broker::INITIALIZING
The peering process is initiated.
- Broker::CONNECTING
Connection establishment in process.
- Broker::CONNECTED
Connection established, peering pending.
- Broker::PEERED
Successfully peered.
- Broker::DISCONNECTED
Connection to remote peer lost.
- Broker::RECONNECTING
Reconnecting to peer after a lost connection.
The possible states of a peer endpoint.
- Broker::TableItem
- Type
-
key:
Broker::Data
val:
Broker::Data
Opaque communication data used as a convenient way to wrap key-value pairs that comprise table entries.
Functions
- Broker::auto_publish
-
Automatically send an event to any interested peers whenever it is locally dispatched. (For example, using “event my_event(…);” in a script.)
- Parameters
topic – a topic string associated with the event message. Peers advertise interest by registering a subscription to some prefix of this topic name.
ev – a Zeek event value.
- Returns
true if automatic event sending is now enabled.
- Broker::auto_unpublish
-
Stop automatically sending an event to peers upon local dispatch.
- Parameters
topic – a topic originally given to
Broker::auto_publish
.ev – an event originally given to
Broker::auto_publish
.
- Returns
true if automatic events will not occur for the topic/event pair.
- Broker::default_log_topic
-
The default implementation for
Broker::log_topic
.
- Broker::flush_logs
-
Sends all pending log messages to remote peers. This normally doesn’t need to be used except for test cases that are time-sensitive.
- Broker::forward
-
Register a topic prefix subscription for events that should only be forwarded to any subscribing peers and not raise any event handlers on the receiving/forwarding node. i.e. it’s the same as
Broker::subscribe
except matching events are not raised on the receiver, just forwarded. UseBroker::unsubscribe
with the same argument to undo this operation.- Parameters
topic_prefix – a prefix to match against remote message topics. e.g. an empty prefix matches everything and “a” matches “alice” and “amy” but not “bob”.
- Returns
true if a new event forwarding/subscription is now registered.
- Broker::listen
- Type
function
(a:string
&default
=Broker::default_listen_address
&optional
, p:port
&default
=Broker::default_port
&optional
, retry:interval
&default
=Broker::default_listen_retry
&optional
) :port
Listen for remote connections using the native Broker protocol.
- Parameters
a – an address string on which to accept connections, e.g. “127.0.0.1”. An empty string refers to INADDR_ANY.
p – the TCP port to listen on. The value 0 means that the OS should choose the next available free port.
retry – If non-zero, retries listening in regular intervals if the port cannot be acquired immediately. 0 disables retries. If the ZEEK_DEFAULT_LISTEN_RETRY environment variable is set (as number of seconds), it overrides any value given here.
- Returns
the bound port or 0/? on failure.
See also:
Broker::status
- Broker::listen_websocket
- Type
function
(a:string
&default
=Broker::default_listen_address_websocket
&optional
, p:port
&default
=Broker::default_port_websocket
&optional
, retry:interval
&default
=Broker::default_listen_retry
&optional
) :port
Listen for remote connections using WebSocket.
- Parameters
a – an address string on which to accept connections, e.g. “127.0.0.1”. An empty string refers to INADDR_ANY.
p – the TCP port to listen on. The value 0 means that the OS should choose the next available free port.
retry – If non-zero, retries listening in regular intervals if the port cannot be acquired immediately. 0 disables retries. If the ZEEK_DEFAULT_LISTEN_RETRY environment variable is set (as number of seconds), it overrides any value given here.
- Returns
the bound port or 0/? on failure.
See also:
Broker::status
- Broker::log_topic
-
A function that will be called for each log entry to determine what broker topic string will be used for sending it to peers. The default implementation will return a value based on
Broker::default_log_topic_prefix
.- Parameters
id – the ID associated with the log stream entry that will be sent.
path – the path to which the log stream entry will be output.
- Returns
a string representing the broker topic to which the log will be sent.
- Broker::node_id
-
Get a unique identifier for the local broker endpoint.
- Returns
a unique identifier for the local broker endpoint.
- Broker::peer
- Type
function
(a:string
, p:port
&default
=Broker::default_port
&optional
, retry:interval
&default
=Broker::default_connect_retry
&optional
) :bool
Initiate a remote connection.
- Parameters
a – an address to connect to, e.g. “localhost” or “127.0.0.1”.
p – the TCP port on which the remote side is listening.
retry – an interval at which to retry establishing the connection with the remote peer if it cannot be made initially, or if it ever becomes disconnected. If the ZEEK_DEFAULT_CONNECT_RETRY environment variable is set (as number of seconds), it overrides any value given here.
- Returns
true if it’s possible to try connecting with the peer and it’s a new peer. The actual connection may not be established until a later point in time.
See also:
Broker::status
- Broker::peers
- Type
function
() :vector
ofBroker::PeerInfo
Get a list of all peer connections.
- Returns
a list of all peer connections.
- Broker::publish_id
-
Publishes the value of an identifier to a given topic. The subscribers will update their local value for that identifier on receipt.
- Parameters
topic – a topic associated with the message.
id – the identifier to publish.
- Returns
true if the message is sent.
- Broker::subscribe
-
Register interest in all peer event messages that use a certain topic prefix. Note that subscriptions may not be altered immediately after calling (except during
zeek_init
).- Parameters
topic_prefix – a prefix to match against remote message topics. e.g. an empty prefix matches everything and “a” matches “alice” and “amy” but not “bob”.
- Returns
true if it’s a new event subscription and it is now registered.
- Broker::unpeer
-
Remove a remote connection.
Note that this does not terminate the connection to the peer, it just means that we won’t exchange any further information with it unless peering resumes later.
- Parameters
a – the address used in previous successful call to
Broker::peer
.p – the port used in previous successful call to
Broker::peer
.TODO – We do not have a function yet to terminate a connection.
- Returns
true if the arguments match a previously successful call to
Broker::peer
.
- Broker::unsubscribe
-
Unregister interest in all peer event messages that use a topic prefix. Note that subscriptions may not be altered immediately after calling (except during
zeek_init
).- Parameters
topic_prefix – a prefix previously supplied to a successful call to
Broker::subscribe
orBroker::forward
.- Returns
true if interest in the topic prefix is no longer advertised.