5. Python Bindings

Almost all functionality of Broker is also accessible through Python bindings. The Python API mostly mimics the C++ interface, but adds transparent conversion between Python values and Broker values. In the following we demonstrate the main parts of the Python API, assuming a general understanding of Broker’s concepts and the C++ interface.

Note

Broker’s Python bindings require Python 2.7 or Python 3. If you are using Python 2.7, then you will need to install the ipaddress module from PyPI (one way to do this is to run “pip install ipaddress”).

5.1. Installation in a Virtual Environment

To install Broker’s python bindings in a virtual environment, the python-prefix configuration option can be specified and the python header files must be on the system for the version of python in the virtual environment. You can also use the prefix configuration option to install the main Broker library and headers into an isolated location.

$ virtualenv -p python3 /Users/user/sandbox/broker/venv
$ . /Users/user/sandbox/broker/venv/bin/activate
$ ./configure --prefix=/Users/user/sandbox/broker --python-prefix=$(python -c 'import sys; print(sys.exec_prefix)')
$ make install
$ python -c 'import broker; print(broker.__file__)'
/Users/user/sandbox/broker/venv/lib/python3.7/site-packages/broker/__init__.py

5.2. Communication

Just as in C++, you first set up peerings between endpoints and create subscriber for the topics of interest:

        ep1 = broker.Endpoint()
        ep2 = broker.Endpoint()

        s1 = ep1.make_subscriber("/test")
        s2 = ep2.make_subscriber("/test")

        port = ep1.listen("127.0.0.1", 0)
        ep2.peer("127.0.0.1", port, 1.0)

You can then start publishing messages. In Python a message is just a list of values, along with the corresponding topic. The following publishes a simple message consisting of just one string, and then has the receiving endpoint wait for it to arrive:

        ep2.publish("/test", ["ping"])
        (t, d) = s1.get()
        # t == "/test", d == ["ping"]

Example of publishing a small batch of two slightly more complex messages with two separate topics:

        msg1 = ("/test/2", (1, 2, 3))
        msg2 = ("/test/3", (42, "foo", {"a": "A", "b": ipaddress.IPv4Address('1.2.3.4')}))
        ep2.publish_batch(msg1, msg2)

As you see with the 2nd message there, elements can be either standard Python values or instances of Broker wrapper classes; see the data model section below for more.

The subscriber instances have more methods matching their C++ equivalent, including available for checking for pending messages, poll() for getting available messages without blocking, fd() for retrieving a select-able file descriptor, and {add,remove}_topic for changing the subscription list.

5.3. Exchanging Zeek Events

The Broker Python bindings come with support for representing Zeek events as well. Here’s the Python version of the C++ ping example shown earlier:

# ping.zeek

redef exit_only_after_terminate = T;

global pong: event(n: int);

event ping(n: int)
	{
	event pong(n);
	}

event zeek_init()
	{
	Broker::subscribe("/topic/test");
	Broker::listen("127.0.0.1", 9999/tcp);
	Broker::auto_publish("/topic/test", pong);
	}
# ping.py

import sys
import broker

# Setup endpoint and connect to Zeek.
ep = broker.Endpoint()
sub = ep.make_subscriber("/topic/test")
ss = ep.make_status_subscriber(True);
ep.peer("127.0.0.1", 9999)

# Wait until connection is established.
st = ss.get()

if not (type(st) == broker.Status and st.code() == broker.SC.PeerAdded):
    print("could not connect")
    sys.exit(0)

for n in range(5):
    # Send event "ping(n)".
    ping = broker.zeek.Event("ping", n);
    ep.publish("/topic/test", ping);

    # Wait for "pong" reply event.
    (t, d) = sub.get()
    pong = broker.zeek.Event(d)
    print("received {}{}".format(pong.name(), pong.args()))
# python3 ping.py
received pong[0]
received pong[1]
received pong[2]
received pong[3]
received pong[4]

5.4. Data Model

The Python API can represent the same type model as the C++ code. For all Broker types that have a direct mapping to a Python type, conversion is handled transparently as values are passed into, or retrieved from, Broker. For example, the message [1, 2, 3] above is automatically converted into a Broker list of three Broker integer values. In cases where there is not a direct Python equivalent for a Broker type (e.g., for count; Python does not have an unsigned integer class), the Broker module provides wrapper classes. The following table summarizes how Broker and Python values are mapped to each other:

Broker Type Python representation
boolean True/False
count broker.Count(x)
integer int
real float
timespan datetime.timedelta
timestamp datetime.datetime
string str
address ipaddress.IPv4Address/ipaddress.IPv6Address
subnet ipaddress.IPv4Network/ipaddress.IPv6Network
port broker.Port(x, broker.Port.{TCP,UDP,ICMP,Unknown})
vector tuple
set set
table dict

Note that either a Python tuple or Python list may convert to a Broker vector, but the canonical Python type representing a vector is a tuple. That is, whenever converting a Broker vector value into a Python value, you will get a tuple. A tuple is the canonical type here because it is an immutable type, but a list is mutable – we need to be able to represent tables indexed by vectors, tables are mapped to Python dictionaries, Python dictionaries only allow immutable index types, and so we must use a tuple to represent a vector.

5.5. Status and Error Messages

Status and error handling works through a status subscriber, again similar to the C++ interface:

        ep1 = broker.Endpoint()
        es1 = ep1.make_status_subscriber()
        r = ep1.peer("127.0.0.1", 1947, 0.0) # Try unavailable port, no retry
        st1 = es1.get()
        # s1.code() == broker.EC.PeerUnavailable
        ep1 = broker.Endpoint()
        ep2 = broker.Endpoint()
        es1 = ep1.make_status_subscriber(True)
        es2 = ep2.make_status_subscriber(True)
        port = ep1.listen("127.0.0.1", 0)
        ep2.peer("127.0.0.1", port, 1.0)
        st1 = es1.get()
        st2 = es2.get()
        # st1.code() == broker.SC.PeerAdded, st2.code() == broker.SC.PeerAdded

5.6. Data Stores

For data stores, the C++ API also directly maps to Python. The following instantiates a master store to then operate on:

        ep1 = broker.Endpoint()
        m = ep1.attach_master("test", broker.Backend.Memory)
        m.put("key", "value")
        x = m.get("key")
        # x == "value"

In Python, both master and clone stores provide all the same accessor and mutator methods as C++. Some examples:

        m.increment("e", 1)
        m.decrement("f", 1)
        m.append("str", "ar")
        m.insert_into("set", 3)
        m.remove_from("set", 1)
        m.insert_into("table", 3, "D")
        m.remove_from("table", 1)
        m.push("vec", 3)
        m.push("vec", 4)
        m.pop("vec")

Here’s a more complete example of using a SQLite-backed data store from python, with the database being stored in mystore.sqlite:

# sqlite-listen.py

import broker

ep = broker.Endpoint()
s = ep.make_subscriber('/test')
ss = ep.make_status_subscriber(True);
ep.listen('127.0.0.1', 9999)

m = ep.attach_master('mystore',
                     broker.Backend.SQLite, {'path': 'mystore.sqlite'})

while True:
    print(ss.get())
    print(m.get('foo'))
# sqlite-connect.py

import broker
import sys
import time

ep = broker.Endpoint()
s = ep.make_subscriber('/test')
ss = ep.make_status_subscriber(True);
ep.peer('127.0.0.1', 9999, 1.0)

st = ss.get();

if not (type(st) == broker.Status and st.code() == broker.SC.PeerAdded):
    print('could not connect')
    sys.exit(1)

c = ep.attach_clone('mystore')

while True:
    time.sleep(1)
    c.increment('foo', 1)
    print(c.get('foo'))