base/frameworks/broker/store.zeek¶
-
Broker
¶
The Broker-based data store API and its various options.
- Namespace
Broker
- Imports
base/bif/data.bif.zeek, base/bif/store.bif.zeek, base/frameworks/broker/main.zeek
Summary¶
Redefinable Options¶
|
The maximum amount of time that a disconnected clone will buffer data store mutation commands. |
The default frequency at which clones will attempt to reconnect/resynchronize with their master in the event that they become disconnected. |
|
The duration after which a clone that is disconnected from its master will begin to treat its local cache as stale. |
|
The directory used for storing persistent database files when using Broker store backed Zeek tables. |
|
If set to true, the current node is the master node for Broker stores backing Zeek tables. |
Types¶
Options to tune the particular storage backends. |
|
Enumerates the possible storage backends. |
|
The result of a data store query. |
|
Whether a data store query could be completed or not. |
|
Options to tune the SQLite storage backend. |
Functions¶
Extends an existing string with another. |
|
Deletes all of a store’s content, it will be empty afterwards. |
|
Close a data store. |
|
Create a clone of a master data store which may live with a remote peer. |
|
Create a master data store which contains key-value pairs. |
|
Convert any Zeek value to communication data. |
|
Retrieve the type of data associated with communication data. |
|
Decrements an existing value by a given amount. |
|
Remove a key-value pair from the store. |
|
Check if a key exists in a data store. |
|
Lookup the value associated with a key in a data store. |
|
Retrieve a specific index from an existing container value. |
|
Increments an existing value by a given amount. |
|
Inserts an element into an existing set. |
|
Inserts an element into an existing table. |
|
Check if a store is closed or not. |
|
Returns a set with all of a store’s keys. |
|
Removes the last element of an existing vector. |
|
Appends an element to an existing vector. |
|
Insert a key-value pair in to the store. |
|
Insert a key-value pair in to the store, but only if the key does not already exist. |
|
Replace a field in a record at a particular position. |
|
Create communication data of type “record”. |
|
Create an iterator for a record. |
|
Check if there are no more elements to iterate over. |
|
Advance an iterator. |
|
Retrieve the data at an iterator’s current position. |
|
Lookup a field in a record at a particular position. |
|
Get the number of fields within a record. |
|
Removes an element from an existing set or table. |
|
Remove all elements within a set. |
|
Check if a set contains a particular element. |
|
Create communication data of type “set”. |
|
Insert an element into a set. |
|
Create an iterator for a set. |
|
Check if there are no more elements to iterate over. |
|
Advance an iterator. |
|
Retrieve the data at an iterator’s current position. |
|
Remove an element from a set. |
|
Get the number of elements within a set. |
|
Get the name of a store. |
|
Remove all elements within a table. |
|
Check if a table contains a particular key. |
|
Create communication data of type “table”. |
|
Insert a key-value pair into a table. |
|
Create an iterator for a table. |
|
Check if there are no more elements to iterate over. |
|
Advance an iterator. |
|
Retrieve the data at an iterator’s current position. |
|
Retrieve a value from a table. |
|
Remove a key-value pair from a table. |
|
Get the number of elements within a table. |
|
Remove all elements within a vector. |
|
Create communication data of type “vector”. |
|
Insert an element into a vector at a particular position, possibly displacing existing elements (insertion always grows the size of the vector by one). |
|
Create an iterator for a vector. |
|
Check if there are no more elements to iterate over. |
|
Advance an iterator. |
|
Retrieve the data at an iterator’s current position. |
|
Lookup an element in a vector at a particular position. |
|
Remove an element from a vector at a particular position. |
|
Replace an element in a vector at a particular position. |
|
Get the number of elements within a vector. |
Detailed Interface¶
Redefinable Options¶
- Broker::default_clone_mutation_buffer_interval¶
-
The maximum amount of time that a disconnected clone will buffer data store mutation commands. If the clone reconnects before this time, it will replay all stored commands. Note that this doesn’t completely prevent the loss of store updates: all mutation messages are fire-and-forget and not explicitly acknowledged by the master. A negative/zero value indicates to never buffer commands.
- Broker::default_clone_resync_interval¶
-
The default frequency at which clones will attempt to reconnect/resynchronize with their master in the event that they become disconnected.
- Broker::default_clone_stale_interval¶
-
The duration after which a clone that is disconnected from its master will begin to treat its local cache as stale. In the stale state, queries to the cache will timeout. A negative value indicates that the local cache is never treated as stale.
- Broker::table_store_db_directory¶
-
The directory used for storing persistent database files when using Broker store backed Zeek tables.
- Broker::table_store_master¶
-
If set to true, the current node is the master node for Broker stores backing Zeek tables. By default this value will be automatically set to true in standalone mode, and on the manager node of a cluster. This value should not typically be changed manually.
Types¶
- Broker::BackendOptions¶
- Type
-
sqlite:
Broker::SQLiteOptions
&default
=[path=]
&optional
Options to tune the particular storage backends.
- Broker::BackendType¶
-
Enumerates the possible storage backends.
- Broker::QueryResult¶
- Type
-
- status:
Broker::QueryStatus
Whether the query completed or not.
- result:
Broker::Data
The result of the query. Certain queries may use a particular data type (e.g. querying store size always returns a count, but a lookup may return various data types).
- status:
The result of a data store query.
- Broker::QueryStatus¶
-
Whether a data store query could be completed or not.
Functions¶
- Broker::append¶
- Type
function
(h:opaque
of Broker::Store, k:any
, s:string
, e:interval
&default
=0 secs
&optional
) :bool
Extends an existing string with another.
- H
the handle of the store to modify.
- K
the key whose associated value is to be modified. The key must already exist.
- S
the string to append.
- E
the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
- Returns
false if the store handle was not valid.
- Broker::clear¶
-
Deletes all of a store’s content, it will be empty afterwards.
- Returns
false if the store handle was not valid.
- Broker::close¶
-
Close a data store.
- H
a data store handle.
- Returns
true if store was valid and is now closed. The handle can no longer be used for data store operations.
- Broker::create_clone¶
- Type
function
(name:string
, resync_interval:interval
&default
=Broker::default_clone_resync_interval
&optional
, stale_interval:interval
&default
=Broker::default_clone_stale_interval
&optional
, mutation_buffer_interval:interval
&default
=Broker::default_clone_mutation_buffer_interval
&optional
) :opaque
of Broker::Store
Create a clone of a master data store which may live with a remote peer. A clone automatically synchronizes to the master by receiving modifications and applying them locally. Direct modifications are not possible, they must be sent through the master store, which then automatically broadcasts the changes out to clones. But queries may be made directly against the local cloned copy, which may be resolved quicker than reaching out to a remote master store.
- Name
the unique name which identifies the master data store.
- Resync_interval
the frequency at which a clone that is disconnected from its master attempts to reconnect with it.
- Stale_interval
the duration after which a clone that is disconnected from its master will begin to treat its local cache as stale. In this state, queries to the clone will timeout. A negative value indicates that the local cache is never treated as stale.
- Mutation_buffer_interval
the amount of time to buffer data store update messages once a clone detects its master is unavailable. If the clone reconnects before this time, it will replay all buffered commands. Note that this doesn’t completely prevent the loss of store updates: all mutation messages are fire-and-forget and not explicitly acknowledged by the master. A negative/zero value indicates that commands never buffer.
- Returns
a handle to the data store for which a subsequent call to
Broker::is_closed
will return true if the store could not be created/opened.
- Broker::create_master¶
- Type
function
(name:string
, b:Broker::BackendType
&default
=Broker::MEMORY
&optional
, options:Broker::BackendOptions
&default
=[sqlite=[path=]]
&optional
) :opaque
of Broker::Store
Create a master data store which contains key-value pairs.
- Name
a unique name for the data store.
- B
the storage backend to use.
- Options
tunes how some storage backends operate.
- Returns
a handle to the data store for which a subsequent call to
Broker::is_closed
will return true if the store could not be created/opened.
- Broker::data¶
- Type
function
(d:any
) :Broker::Data
Convert any Zeek value to communication data.
Note
Normally you won’t need to use this function as data conversion happens implicitly when passing Zeek values into Broker functions.
- D
any Zeek value to attempt to convert (not all types are supported).
- Returns
the converted communication data. If the supplied Zeek data type does not support conversion to communication data, the returned record’s optional field will not be set.
- Broker::data_type¶
- Type
function
(d:Broker::Data
) :Broker::DataType
Retrieve the type of data associated with communication data.
- D
the communication data.
- Returns
The data type associated with the communication data. Note that Broker represents records in the same way as vectors, so there is no “record” type.
- Broker::decrement¶
- Type
function
(h:opaque
of Broker::Store, k:any
, a:any
&default
=1
&optional
, e:interval
&default
=0 secs
&optional
) :bool
Decrements an existing value by a given amount. This is supported for all numerical types, as well as for timestamps.
- H
the handle of the store to modify.
- K
the key whose associated value is to be modified. The key must already exist.
- Amount
the amount to decrement the value by.
- E
the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
- Returns
false if the store handle was not valid.
- Broker::erase¶
-
Remove a key-value pair from the store.
- H
the handle of the store to modify.
- K
the key to remove.
- Returns
false if the store handle was not valid.
- Broker::exists¶
- Type
function
(h:opaque
of Broker::Store, k:any
) :Broker::QueryResult
Check if a key exists in a data store.
- H
the handle of the store to query.
- K
the key to lookup.
- Returns
True if the key exists in the data store.
- Broker::get¶
- Type
function
(h:opaque
of Broker::Store, k:any
) :Broker::QueryResult
Lookup the value associated with a key in a data store.
- H
the handle of the store to query.
- K
the key to lookup.
- Returns
the result of the query.
- Broker::get_index_from_value¶
- Type
function
(h:opaque
of Broker::Store, k:any
, i:any
) :Broker::QueryResult
Retrieve a specific index from an existing container value. This is supported for values of types set, table, and vector.
- H
the handle of the store to query.
- K
the key of the container value to lookup.
- I
the index to retrieve from the container value.
- Returns
For tables and vectors, the value at the given index, or failure if the index doesn’t exist. For sets, a boolean indicating whether the index exists. Returns failure if the key does not exist at all.
- Broker::increment¶
- Type
function
(h:opaque
of Broker::Store, k:any
, a:any
&default
=1
&optional
, e:interval
&default
=0 secs
&optional
) :bool
Increments an existing value by a given amount. This is supported for all numerical types, as well as for timestamps.
- H
the handle of the store to modify.
- K
the key whose associated value is to be modified. The key must already exist.
- A
the amount to increment the value by.
- E
the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
- Returns
false if the store handle was not valid.
- Broker::insert_into_set¶
- Type
function
(h:opaque
of Broker::Store, k:any
, i:any
, e:interval
&default
=0 secs
&optional
) :bool
Inserts an element into an existing set.
- H
the handle of the store to modify.
- K
the key whose associated value is to be modified. The key must already exist.
- I
the index to insert into the set.
- E
the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
- Returns
false if the store handle was not valid.
- Broker::insert_into_table¶
- Type
function
(h:opaque
of Broker::Store, k:any
, i:any
, v:any
, e:interval
&default
=0 secs
&optional
) :bool
Inserts an element into an existing table.
- H
the handle of the store to modify.
- K
the key whose associated value is to be modified. The key must already exist.
- I
the index to insert into the table
- V
the value to associate with the index.
- E
the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
- Returns
false if the store handle was not valid.
- Broker::is_closed¶
-
Check if a store is closed or not.
- Returns
true if the store is closed.
- Broker::keys¶
- Type
function
(h:opaque
of Broker::Store) :Broker::QueryResult
Returns a set with all of a store’s keys. The results reflect a snapshot in time that may diverge from reality soon afterwards. When acessing any of the element, it may no longer actually be there. The function is also expensive for large stores, as it copies the complete set.
- Returns
a set with the keys. If you expect the keys to be of non-uniform type, consider using
Broker::set_iterator
to iterate over the result.
- Broker::pop¶
-
Removes the last element of an existing vector.
- H
the handle of the store to modify.
- K
the key whose associated value is to be modified. The key must already exist.
- E
the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
- Returns
false if the store handle was not valid.
- Broker::push¶
- Type
function
(h:opaque
of Broker::Store, k:any
, v:any
, e:interval
&default
=0 secs
&optional
) :bool
Appends an element to an existing vector.
- H
the handle of the store to modify.
- K
the key whose associated value is to be modified. The key must already exist.
- B
the value to append to the vector.
- E
the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
- Returns
false if the store handle was not valid.
- Broker::put¶
- Type
function
(h:opaque
of Broker::Store, k:any
, v:any
, e:interval
&default
=0 secs
&optional
) :bool
Insert a key-value pair in to the store.
- H
the handle of the store to modify.
- K
the key to insert.
- V
the value to insert.
- E
the expiration interval of the key-value pair.
- Returns
false if the store handle was not valid.
- Broker::put_unique¶
- Type
function
(h:opaque
of Broker::Store, k:any
, v:any
, e:interval
&default
=0 secs
&optional
) :Broker::QueryResult
Insert a key-value pair in to the store, but only if the key does not already exist.
- H
the handle of the store to modify.
- K
the key to insert.
- V
the value to insert.
- E
the expiration interval of the key-value pair.
- Returns
the result of the query which is a boolean data value that is true if the insertion happened, or false if it was rejected due to the key already existing.
- Broker::record_assign¶
- Type
function
(r:Broker::Data
, idx:count
, d:any
) :bool
Replace a field in a record at a particular position.
- R
the record to modify.
- D
the new field value to assign.
- Idx
the index to replace.
- Returns
false if the index was larger than any valid index, else true.
- Broker::record_create¶
- Type
function
(sz:count
) :Broker::Data
Create communication data of type “record”.
- Sz
the number of fields in the record.
- Returns
record data, with all fields uninitialized.
- Broker::record_iterator¶
- Type
function
(r:Broker::Data
) :opaque
of Broker::RecordIterator
Create an iterator for a record. Note that this makes a copy of the record internally to ensure the iterator is always valid.
- R
the record to iterate over.
- Returns
an iterator.
- Broker::record_iterator_last¶
-
Check if there are no more elements to iterate over.
- It
an iterator.
- Returns
true if there are no more elements to iterator over, i.e. the iterator is one-past-the-final-element.
- Broker::record_iterator_next¶
-
Advance an iterator.
- It
an iterator.
- Returns
true if the iterator, after advancing, still references an element in the collection. False if the iterator, after advancing, is one-past-the-final-element.
- Broker::record_iterator_value¶
- Type
function
(it:opaque
of Broker::RecordIterator) :Broker::Data
Retrieve the data at an iterator’s current position.
- It
an iterator.
- Returns
element in the collection that the iterator currently references.
- Broker::record_lookup¶
- Type
function
(r:Broker::Data
, idx:count
) :Broker::Data
Lookup a field in a record at a particular position.
- R
the record to query.
- Idx
the index to lookup.
- Returns
the value at the index. The optional field of the returned record may not be set if the field of the record has no value or if the index was not valid.
- Broker::record_size¶
- Type
function
(r:Broker::Data
) :count
Get the number of fields within a record.
- R
the record to query.
- Returns
the number of fields in the record.
- Broker::remove_from¶
- Type
function
(h:opaque
of Broker::Store, k:any
, i:any
, e:interval
&default
=0 secs
&optional
) :bool
Removes an element from an existing set or table.
- H
the handle of the store to modify.
- K
the key whose associated value is to be modified. The key must already exist.
- I
the index to remove from the set or table.
- E
the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
- Returns
false if the store handle was not valid.
- Broker::set_clear¶
- Type
function
(s:Broker::Data
) :bool
Remove all elements within a set.
- S
the set to clear.
- Returns
always true.
- Broker::set_contains¶
- Type
function
(s:Broker::Data
, key:any
) :bool
Check if a set contains a particular element.
- S
the set to query.
- Key
the element to check for existence.
- Returns
true if the key exists in the set.
- Broker::set_create¶
- Type
function
() :Broker::Data
Create communication data of type “set”.
- Broker::set_insert¶
- Type
function
(s:Broker::Data
, key:any
) :bool
Insert an element into a set.
- S
the set to modify.
- Key
the element to insert.
- Returns
true if the key was inserted, or false if it already existed.
- Broker::set_iterator¶
- Type
function
(s:Broker::Data
) :opaque
of Broker::SetIterator
Create an iterator for a set. Note that this makes a copy of the set internally to ensure the iterator is always valid.
- S
the set to iterate over.
- Returns
an iterator.
- Broker::set_iterator_last¶
-
Check if there are no more elements to iterate over.
- It
an iterator.
- Returns
true if there are no more elements to iterator over, i.e. the iterator is one-past-the-final-element.
- Broker::set_iterator_next¶
-
Advance an iterator.
- It
an iterator.
- Returns
true if the iterator, after advancing, still references an element in the collection. False if the iterator, after advancing, is one-past-the-final-element.
- Broker::set_iterator_value¶
- Type
function
(it:opaque
of Broker::SetIterator) :Broker::Data
Retrieve the data at an iterator’s current position.
- It
an iterator.
- Returns
element in the collection that the iterator currently references.
- Broker::set_remove¶
- Type
function
(s:Broker::Data
, key:any
) :bool
Remove an element from a set.
- S
the set to modify.
- Key
the element to remove.
- Returns
true if the element existed in the set and is now removed.
- Broker::set_size¶
- Type
function
(s:Broker::Data
) :count
Get the number of elements within a set.
- S
the set to query.
- Returns
the number of elements in the set.
- Broker::store_name¶
-
Get the name of a store.
- Returns
the name of the store.
- Broker::table_clear¶
- Type
function
(t:Broker::Data
) :bool
Remove all elements within a table.
- T
the table to clear.
- Returns
always true.
- Broker::table_contains¶
- Type
function
(t:Broker::Data
, key:any
) :bool
Check if a table contains a particular key.
- T
the table to query.
- Key
the key to check for existence.
- Returns
true if the key exists in the table.
- Broker::table_create¶
- Type
function
() :Broker::Data
Create communication data of type “table”.
- Broker::table_insert¶
- Type
function
(t:Broker::Data
, key:any
, val:any
) :Broker::Data
Insert a key-value pair into a table.
- T
the table to modify.
- Key
the key at which to insert the value.
- Val
the value to insert.
- Returns
true if the key-value pair was inserted, or false if the key already existed in the table.
- Broker::table_iterator¶
- Type
function
(t:Broker::Data
) :opaque
of Broker::TableIterator
Create an iterator for a table. Note that this makes a copy of the table internally to ensure the iterator is always valid.
- T
the table to iterate over.
- Returns
an iterator.
- Broker::table_iterator_last¶
-
Check if there are no more elements to iterate over.
- It
an iterator.
- Returns
true if there are no more elements to iterator over, i.e. the iterator is one-past-the-final-element.
- Broker::table_iterator_next¶
-
Advance an iterator.
- It
an iterator.
- Returns
true if the iterator, after advancing, still references an element in the collection. False if the iterator, after advancing, is one-past-the-final-element.
- Broker::table_iterator_value¶
- Type
function
(it:opaque
of Broker::TableIterator) :Broker::TableItem
Retrieve the data at an iterator’s current position.
- It
an iterator.
- Returns
element in the collection that the iterator currently references.
- Broker::table_lookup¶
- Type
function
(t:Broker::Data
, key:any
) :Broker::Data
Retrieve a value from a table.
- T
the table to query.
- Key
the key to lookup.
- Returns
the value associated with the key. If the key did not exist, then the optional field of the returned record is not set.
- Broker::table_remove¶
- Type
function
(t:Broker::Data
, key:any
) :Broker::Data
Remove a key-value pair from a table.
- T
the table to modify.
- Key
the key to remove from the table.
- Returns
the value associated with the key. If the key did not exist, then the optional field of the returned record is not set.
- Broker::table_size¶
- Type
function
(t:Broker::Data
) :count
Get the number of elements within a table.
- T
the table to query.
- Returns
the number of elements in the table.
- Broker::vector_clear¶
- Type
function
(v:Broker::Data
) :bool
Remove all elements within a vector.
- V
the vector to clear.
- Returns
always true.
- Broker::vector_create¶
- Type
function
() :Broker::Data
Create communication data of type “vector”.
- Broker::vector_insert¶
- Type
function
(v:Broker::Data
, idx:count
, d:any
) :bool
Insert an element into a vector at a particular position, possibly displacing existing elements (insertion always grows the size of the vector by one).
- V
the vector to modify.
- D
the element to insert.
- Idx
the index at which to insert the data. If it is greater than the current size of the vector, the element is inserted at the end.
- Returns
always true.
- Broker::vector_iterator¶
- Type
function
(v:Broker::Data
) :opaque
of Broker::VectorIterator
Create an iterator for a vector. Note that this makes a copy of the vector internally to ensure the iterator is always valid.
- V
the vector to iterate over.
- Returns
an iterator.
- Broker::vector_iterator_last¶
-
Check if there are no more elements to iterate over.
- It
an iterator.
- Returns
true if there are no more elements to iterator over, i.e. the iterator is one-past-the-final-element.
- Broker::vector_iterator_next¶
-
Advance an iterator.
- It
an iterator.
- Returns
true if the iterator, after advancing, still references an element in the collection. False if the iterator, after advancing, is one-past-the-final-element.
- Broker::vector_iterator_value¶
- Type
function
(it:opaque
of Broker::VectorIterator) :Broker::Data
Retrieve the data at an iterator’s current position.
- It
an iterator.
- Returns
element in the collection that the iterator currently references.
- Broker::vector_lookup¶
- Type
function
(v:Broker::Data
, idx:count
) :Broker::Data
Lookup an element in a vector at a particular position.
- V
the vector to query.
- Idx
the index to lookup.
- Returns
the value at the index. If the index was larger than any valid index, the optional field of the returned record is not set.
- Broker::vector_remove¶
- Type
function
(v:Broker::Data
, idx:count
) :Broker::Data
Remove an element from a vector at a particular position.
- V
the vector to modify.
- Idx
the index to remove.
- Returns
the value that was just evicted. If the index was larger than any valid index, the optional field of the returned record is not set.
- Broker::vector_replace¶
- Type
function
(v:Broker::Data
, idx:count
, d:any
) :Broker::Data
Replace an element in a vector at a particular position.
- V
the vector to modify.
- D
the element to insert.
- Idx
the index to replace.
- Returns
the value that was just evicted. If the index was larger than any valid index, the optional field of the returned record is not set.
- Broker::vector_size¶
- Type
function
(v:Broker::Data
) :count
Get the number of elements within a vector.
- V
the vector to query.
- Returns
the number of elements in the vector.