Reference
Interface Definitions (“evt files”)
An analyzer for Zeek does more than just parsing data. Accordingly, we need to tell Zeek a couple of additional pieces about Spicy parsers we want it to provide to Zeek:
- Analyzer setup
Zeek needs to know what type of analyzers we are creating, when we want Zeek to activate them, and what Spicy unit types to use as their parsing entry point.
- Event definitions
We need to tell Zeek which events to provide and when to trigger them.
We define all of these through custom interface definition files that
Spicy’s compiler for Zeek reads in. These files use an *.evt
extension, and the following subsections discuss their content in more
detail.
Generally, empty lines and comments starting with #
are ignored in
an *.evt
.
Note
The syntax for *.evt
files comes with some legacy pieces that
aren’t particularly pretty. We may clean that up at some point.
Analyzer Setup
You can define protocol analyzers, packet analyzers and file analyzers in an
*.evt
file, per the following.
Protocol Analyzer
To define a protocol analyzer, add a new section to an *.evt
file that looks like this:
protocol analyzer ANALYZER_NAME over TRANSPORT_PROTOCOL:
PROPERTY_1,
PROPERTY_2,
...
PROPERTY_N;
Here, ANALYZER_NAME
is a name to identify your analyzer inside
Zeek. You can choose names arbitrarily as long as they are unique. As
a convention, however, we recommend name with a spicy::*
prefix
(e.g., spicy::BitTorrent
).
On the Zeek-side, through some normalization, these names
automatically turn into tags added to Zeek’s Analyzer::Tag
enum.
For example, spicy::BitTorrent
turns into
Analyzer::ANALYZER_SPICY_BITTORRENT
.
The analyzer’s name is also what goes into Zeek signatures to activate
an analyzer DPD-style. If the name is spicy::BitTorrent
, you’d
write enable "spicy::BitTorrent"
into the signature.
Note
Once you have made your analyzers available to Zeek (which we will
discuss below), running zeek -NN Zeek::Spicy
will show you a
summary of what’s now available, including their Zeek-side names
and tags.
TRANSPORT_PROTOCOL
can be either tcp
or udp
, depending on
the transport-layer protocol that your new analyzer wants to sit on
top of.
Following that initial protocol analyzer ...
line, a set of
properties defines further specifics of your analyzer. The following
properties are supported:
parse [originator|responder] with SPICY_UNIT
Specifies the top-level Spicy unit(s) the analyzer uses for parsing payload, with
SPICY_UNIT
being a fully-qualified Spicy-side type name (e.g.HTTP::Request
). The unit type must have been declared aspublic
in Spicy.If
originator
is given, the unit is used only for parsing the connection’s originator-side payload; and ifresponder
is given, only for responder-side payload. If neither is given, it’s used for both sides. In other words, you can use different units per side by specifying two propertiesparse originator with ...
andparse responder with ...
.port PORT
orports { PORT_1, ..., PORT_M }
Specifies one or more well-known ports for which you want Zeek to automatically activate your analyzer with corresponding connections. Each port must be specified in Spicy’s syntax for port constants (e.g.,
80/tcp
), or as a port rangePORT_START-PORT_END
where start and end port are port constants forming a closed interval. The ports’ transport protocol must match that of the analyzer.Note
Zeek will also honor any
%port
meta data property that the responder-sideSPICY_UNIT
may define (as long as the attribute’s direction is notoriginator
).Note
While using
port
(or%port
) can be convinient, for production analyzers we recommended to instead register their well-known ports from inside a Zeek script, using a snippet like this:module MyAnalyzer; export { const ports = { 12345/tcp } &redef; } redef likely_server_ports += { ports }; event zeek_init() &priority=5 { Analyzer::register_for_ports(Analyzer::ANALYZER_MY_ANALYZER, ports); }This follows the idiomatic Zeek pattern for defining well-known ports that allows users to customize them through their own site-specific scripts (e.g.,
redef MyAnalyzer::port += { 12346/tcp };
). The package template includes such code instead of defining ports inside the EVT file.replaces ANALYZER_NAME
Replaces a built-in analyzer that Zeek already provides with a new Spicy version by internally disabling the existing analyzer and redirecting any usage to the new Spicy analyzer instead.
ANALYZER_NAME
is the Zeek-side name of the analyzer. To find that name, inspect the output ofzeek -NN
for available analyzers:# zeek -NN | grep '\[Analyzer\]' ... [Analyzer] SMTP (ANALYZER_SMTP, enabled) ...Here,
SMTP
is the name you would write intoreplaces
to disable the built-in SMTP analyzer.The replacement takes effect in most places where normally the existing Zeek analyzer would be used, or gets referenced, including in Zeek scripts (e.g., when registering well-known ports).
Note
replaces
is not limited to substituting built-in analyzers; you could also replace one Spicy analyzer with another. However, there is no support for more complex scenarios like chains of analyzers replacing each other; results are undefined in that case. It’s best to stick to replacing classic, built-in analyzers, and control everything else simply through not loading any undesired Spicy analyzers in the first place.
As a full example, here’s what a new HTTP analyzer could look like:
protocol analyzer spicy::HTTP over TCP:
parse originator with HTTP::Requests,
parse responder with HTTP::Replies,
port 80/tcp,
replaces HTTP;
Packet Analyzer
Defining packet analyzers works quite similar to protocol analyzers through
*.evt
sections like this:
packet analyzer ANALYZER_NAME:
PROPERTY_1,
PROPERTY_2,
...
PROPERTY_N;
Here, ANALYZER_NAME
is again a name to identify your analyzer
inside Zeek. On the Zeek-side, the name will be added to Zeek’s
PacketAnalyzer::Tag
enum.
Packet analyzers support the following properties:
parse with SPICY_UNIT
Specifies the top-level Spicy unit the analyzer uses for parsing each packet, with
SPICY_UNIT
being a fully-qualified Spicy-side type name. The unit type must have been declared aspublic
in Spicy.replaces ANALYZER_NAME
Disables an existing packet analyzer that Zeek already provides internally, allowing you to replace a built-in analyzer with a new Spicy version.
ANALYZER_NAME
is the Zeek-side name of the existing analyzer. To find that name, inspect the output ofzeek -NN
for available analyzers:# zeek -NN | grep '\[Packet Analyzer\]' ... [Packet Analyzer] Ethernet (ANALYZER_ETHERNET) ...Here,
Ethernet
is the name you would give toreplaces
to disable the built-in Ethernet analyzer.When replacing an existing packet analyzer, you still need to configure your new analyzer on the Zeek side through a spicy_init() event handler. See below for more.
Note
This feature requires Zeek >= 5.2.
As a full example, here’s what a new packet analyzer could look like:
packet analyzer spicy::RawLayer:
parse with RawLayer::Packet;
In addition to the Spicy-side configuration, packet analyzers also
need to be registered with Zeek inside a spicy_init
event handler;
see the Zeek documentation
for more. The -NN
output shows your new analyzer’s ANALYZER_*
tag to use.
Note
Depending on relative loading order between your analyzer and
Zeek’s scripts, in some situations you may not have access to the
ANALYZER_*
tag referring to your new analyzer. If you run into
this, use PacketAnalyzer::try_register_packet_analyzer_by_name
to register your Spicy analyzer by name (as shown by -NN
).
Example:
event spicy_init()
{
if ( ! PacketAnalyzer::try_register_packet_analyzer_by_name("Ethernet", 0x88b5, "spicy::RawLayer") )
Reporter::error("unknown analyzer name used");
}
File Analyzer
Defining file analyzers works quite similar to protocol analyzers,
through *.evt
sections like this:
file analyzer ANALYZER_NAME:
PROPERTY_1,
PROPERTY_2,
...
PROPERTY_N;
Here, ANALYZER_NAME
is again a name to identify your analyzer
inside Zeek. On the Zeek-side, the name will be added to Zeek’s
Files::Tag
enum.
File analyzers support the following properties:
parse with SPICY_UNIT
Specifies the top-level Spicy unit the analyzer uses for parsing file content, with
SPICY_UNIT
being a fully-qualified Spicy-side type name. The unit type must have been declared aspublic
in Spicy.mime-type MIME-TYPE
Specifies a MIME type for which you want Zeek to automatically activate your analyzer when it sees a corresponding file on the network. The type is specified in standard
type/subtype
notion, without quotes (e.g.,image/gif
).Note
Zeek will also honor any
%mime-type
meta data property that theSPICY_UNIT
may define.Note
Keep in mind that Zeek identifies MIME types through “content sniffing” (i.e., similar to libmagic), and usually not by protocol-level headers (e.g., not through HTTP’s
Content-Type
header). If in doubt, examinefiles.log
for what it records as a file’s type.replaces ANALYZER_NAME
Disables an existing file analyzer that Zeek already provides internally, allowing you to replace a built-in analyzer with a new Spicy version.
ANALYZER_NAME
is the Zeek-side name of the analyzer. To find that name, inspect the output ofzeek -NN
for available analyzers:# zeek -NN | grep '\[File Analyzer\]' ... [File Analyzer] PE (ANALYZER_PE, enabled) ...Here,
PE
is the name you would write intoreplaces
to disable the built-in PE analyzer.Note
This feature requires Zeek >= 4.1.
As a full example, here’s what a new GIF analyzer could look like:
file analyzer spicy::GIF:
parse with GIF::Image,
mime-type image/gif;
Event Definitions
To define a Zeek event that you want the Spicy analyzer to trigger, you add lines of the form:
on HOOK_ID -> event EVENT_NAME(ARG_1, ..., ARG_N);
on HOOK_ID if COND -> event EVENT_NAME(ARG_1, ..., ARG_N);
Zeek automatically derives from this everything it needs to register new events with Zeek, including a mapping of the arguments’ Spicy types to corresponding Zeek types. More specifically, these are the pieces going into such an event definition:
on HOOK_ID
A Spicy-side ID that defines when you want to trigger the event. This works just like an
on ...
unit hook, and you can indeed use anything here that Spicy supports for those as well (except container hooks). So, e.g.,on HTTP::Request::%done
triggers an event whenever aHTTP::Request
unit has been fully parsed, andon HTTP::Request::uri
leads to an event each time theuri
field has been parsed. (In the former example you may skip the%done
, actually:on HTTP::Request
implicitly adds it.)EVENT_NAME
The Zeek-side name of the event you want to generate, preferably including a namespace (e.g.,
http::request
).ARG_1, ..., ARG_N
Arguments to pass to the event, given as arbitrary Spicy expressions. Each expression will be evaluated within the context of the unit that the
on ...
triggers on, similar to code running inside the body of a corresponding unit hook. That means the expression has access toself
for accessing the unit instance that’s currently being parsed.The Spicy type of the expression determines the Zeek-side type of the corresponding event argument. Most Spicy types translate over pretty naturally, the following summarizes the translation:
Spicy Type
Zeek Type
Notes
addr
addr
bool
bool
bytes
string
enum { ... }
enum { ... }
[1]
int(8|16|32|64)
int
interval
interval
list<T>
vector of T
map<V,K>
table[V] of K
optional<T>
T
[2]
port
port
real
double
set<T>
set[T]
string
string
time
time
tuple<T_1, ... ,T_N>
record { T1, ..., T_N }
[3]
uint(8|16|32|64)
count
vector<T>
vector of T
Note
- [1]
A corresponding Zeek-side
enum
type is automatically created. See below for more.One special-case: For convenience, the Spicy-side Protocol enum is automatically mapped to the Zeek-side transport_proto enum.
- [2]
The optional value must have a value, otherwise a runtime exception will be thrown.
- [3]
Must be mapped to a Zeek-side record type with matching fields.
If a tuple element is mapped to a record field with a
&default
or&optional
attribute, a couple special cases are supported:If the expression evaluates to
Null
, the record field is left unset.If the element’s expression uses the .? operator and that fails to produce a value, the record field is likewise left unset.
In addition to full Spicy expressions, there are four reserved IDs with specific meanings when used as arguments:
$conn
Refers to the connection that’s currently being processed by Zeek. On the Zeek-side this will turn into a parameter of Zeek type
connection
. This ID can be used only with protocol analyzers.$file
Refers to the file that’s currently being processed by Zeek. On the Zeek-side this will turn into a parameter of Zeek type
fa_file
. This ID can be used with file or protocol analyzers. For protocol analyzers it refers to the most recently opened, but not yet closed file, see zeek::file_begin and zeek::file_end.$packet
Refers to the packet that’s currently being processed by Zeek. On the Zeek-side this will turn into a parameter of Zeek type
raw_pkt_hdr
, with any fields filled in that have already been parsed by Zeek’s built-in analyzers. This ID can be used only with packet analyzers. (Note that instantiation ofraw_pkt_hdr
values can be relatively expensive on the Zeek side, so best to limit usage of this ID to a small part of the overall traffic.)$is_orig
A boolean indicating if the data currently being processed is coming from the originator (
True
) or responder (False
) of the underlying connection. This turns into a corresponding boolean value on the Zeek side. This ID can be used only with protocol analyzers.
Note
Some tips:
If you want to force a specific type on the Zeek-side, you have a couple of options:
Spicy may provide a
cast
operator from the actual type into the desired type (e.g.,cast<uint64>(..)
).Argument expressions have access to global functions defined in the Spicy source files, so you can write a conversion function taking an argument with its original type and returning it with the desired type.
List comprehension can be convenient to fill Zeek vectors:
[some_func(i) for i in self.my_list]
.
if COND
If given, events are only generated if the expression
COND
evaluates to true. Just like event arguments, the expression is evaluated in the context of the current unit instance and has access toself
.
Exporting Types
As we discuss above, the type of each event
argument maps over to a corresponding Zeek type. On the Zeek side,
that corresponding type needs to be known by Zeek. That is always the
case for built-in atomic types (per the conversion table), but it can turn out more challenging
to achieve for custom types, such as enum
and record
, for
which you would normally need to create matching type declarations in
your Zeek scripts. While that’s not necessarily hard, but it can
become quite cumbersome.
Fortunately, there’s help: for most types, Zeek can instantiate corresponding types automatically as it loads the corresponding analyzer. While you will never actually see the Zeek-side type declarations, they will be available inside your Zeek scripts as if you had typed them out yourself–similar to other types that are built into Zeek itself.
To have the Zeek create a type for your analyzer automatically,
you need to export
the Spicy type in your EVT file. The syntax for
that is:
export SPICY_ID [as ZEEK_ID];
Here, SPICY_ID
is the fully-scoped type ID on the Spicy side, and
ZEEK_ID
is the fully-scoped type ID you want in Zeek. If you leave
out the as ...
part, the Zeek name will be the same as the Spicy
name, including its namespace. For example, say you have a Spicy unit
TFTP::Request
. Adding export TFTP::Request;
to your EVT file
will make a record
type of the same name, and with the same
fields, available in Zeek. If you instead use export TFTP::Request
as TheOtherTFTP::Request
, it will be placed into a different
namespace instead.
Exporting types generally works for most Spicy types as long as
there’s an ID associated with them in your Spicy code. However,
exporting is most helpful with user-defined types, such as enum
and unit
, because it can save you quite a bit of typing there. We
discuss the more common type conversions in more detail below.
To confirm the types made available to Zeek, you can see all exports
in the output of zeek -NN
. With our 2nd TFTP::Request
example, that looks like this:
[...]
# zeek -NN Zeek::Spicy
Zeek::Spicy - Support for Spicy parsers (*.hlto)
[Type] TheOtherTFTP::Request
[...]
Note
Most, but not all, types can be exported automatically. For
example, self-recursive types are currently not supported.
Generally, if you run into trouble exporting a type, you can always
fall back to declaring a corresponding Zeek version yourself in
your Zeek script. Consider the export
mechanism as a
convenience feature that helps avoid writing a lot of boiler plate
code in common cases.
Enum Types
When you export a Spicy enum
type, Zeek creates a
corresponding Zeek enum
type. For example, assume the following
Spicy declaration:
module Test;
type MyEnum = enum {
A = 83,
B = 84,
C = 85
};
Using export Test::MyEnum;
, Zeek will create the equivalent
of the following Zeek type for use in your scripts:
module Test;
export {
type MyEnum: enum {
MyEnum_A = 83,
MyEnum_B = 84,
MyEnum_A = 85,
MyEnum_Undef = -1
};
}
(The odd naming is due to ID limitations on the Zeek side.)
Note
For backward compatibility, the enum
type comes with an
additional property: all public enum types are automatically
exported, even without adding any export
to your EVT file.
This feature may go away at some point, and we suggest to not rely
on it on new code; always use export
for enum types as well.
Unit Types
When you export a Spicy unit
type, Zeek creates a
corresponding Zeek record
type. For example, assume the following
Spicy declaration:
module Test;
type MyRecord = unit {
a: bytes &size=4;
b: uint16;
var c: bool;
};
Using export Test::MyRecord;
, Zeek will then create the
equivalent of the following Zeek type for use in your scripts:
module Test;
export {
type MyRecord: record {
a: string &optional;
b: count &optional;
c: bool;
};
}
The individual fields map over just like event arguments do, following
the the table above. For aggregate
types, this works recursively: if, e.g., a field is itself of unit
type and that type has been exported as well, Zeek will map it
over accordingly to the corresponding record
type. Note that such
dependent types must be exported first in the EVT file for this to
work. As a result, you cannot export self-recursive unit types.
As you can see in the example, unit fields are always declared as
&optional
on the Zeek-side, as they may not have been set during
parsing. Unit variables are non-optional by default, unless declared
as &optional
in Spicy.
Controlling created Zeek types
If needed the automatic unit type exporting described above can be customized. The general syntax for this is:
export SPICY_ID [with { [SIPCY_FIELD_NAME [&log], ]... }];
export SPICY_ID [as ZEEK_ID [ with { [SIPCY_FIELD_NAME [&log], ]...] }];
export SPICY_ID [without { [SIPCY_FIELD_NAME, ]... }];
export SPICY_ID [as ZEEK_ID [ without { [SIPCY_FIELD_NAME, ]...] }];
export SPICY_ID &log;
export SPICY_ID as ZEEK_ID &log;
This allows
including fields to export by naming them in
with
, e.g.,export foo::A with { x, y }
creates a Zeek record with the Spicy unit fieldsx
andy
added to the Zeek record type.excluding fields from export by naming them in
without
, e.g.,export foo::A without { x }
creates a Zeek record with all fields butx
of the Spicy unit exported.renaming the Spicy type on export, e.g.,
export foo::A as bar::X
exports the Spicy unit typeA
in Spicy modulefoo
to a Zeek record typeX
in Zeek modulebar
.controlling
&log
attribute on generated Zeek record types. We can either the mark all Zeek record fields&log
by marking the whole type&log
with e.g.,export foo::A &log
orexport foo::A as bar::X &log
, or mark individual fields&log
in thewith
field list.
Example: Controlling exported fields
Assume we are given a Spicy unit type foo::X
:
module foo;
public type X = unit {
x: uint8;
y: uint8;
z: uint8;
};
To create a Zeek type without the field x
we can use the following export
statement:
export foo::X without { x };
Example: Adding Zeek &log
attributes to created Zeek record types
In Zeek records, data intended to be written to Zeek log streams needs to be
marked &log
. We can trigger creation of this attribute from Spicy
either per-field or for the whole record
.
The export statement
export foo::X &log;
creates a Zeek record type
module foo;
export {
type X: record {
x: count &optional &log;
y: count &optional &log;
z: count &optional &log;
};
}
To mark individual fields &log
we can use the attribute on the
respective fields, e.g.,
export foo::X with { x &log, y, z };
creates a Zeek record type
module foo;
export {
type X: record {
x: count &optional &log;
y: count &optional;
z: count &optional;
};
}
Struct Types
A Spicy struct
type maps over to Zeek in the same way as unit
types do, treating each field like a unit variable. See there for more information.
Tuple Types
A Spicy tuple
type maps over to a Zeek record
similar to how
unit
types do, treating each tuple element like a unit variable.
See there for more information.
Exporting works only for tuple
types that declare names for all
their elements.
Importing Spicy Modules
Code in an *.evt
file may need access to additional Spicy modules,
such as when expressions for event parameters call Spicy
functions defined elsewhere. To make a Spicy module available, you can
insert import
statements into the *.evt
file that work
just like in Spicy code:
import NAME
Imports Spicy module
NAME
.import NAME from X.Y.Z;
Searches for the module
NAME
(i.e., for the filenameNAME.spicy
) inside a sub-directoryX/Y/Z
along the search path, and then imports it.
Conditional Compilation
*.evt
files offer the same basic form of conditional
compilation through
@if
/@else
/@endif
blocks as Spicy scripts. Zeek
makes two additional identifiers available for testing to both
*.evt
and *.spicy
code:
HAVE_ZEEK
Always set to 1 by Zeek. This can be used for feature testing from Spicy code to check if it’s being compiled for Zeek.
ZEEK_VERSION
The numerical Zeek version that’s being compiled for (see
zeek -e 'print Version::number'
).
This is an example bracketing code by Zeek version in an EVT file:
@if ZEEK_VERSION < 30200
<EVT code for Zeek versions older than 3.2>
@else
<EVT code for Zeek version 3.2 or newer>
@endif
Compiling Analyzers
Once you have the *.spicy
and *.evt
source files for your new
analyzer, you need to precompile them into an *.hlto
object file
containing their final executable code, which Zeek will then use. To
do that, pass the relevant *.spicy
and *.evt
files to
spicyz
, then have Zeek load the output. To repeat the
example from the Getting Started
guide:
# spicyz -o my-http-analyzer.hlto my-http.spicy my-http.evt
# zeek -Cr request-line.pcap my-http-analyzer.hlto my-http.zeek
Zeek saw from 127.0.0.1: GET /index.html 1.0
Instead of providing the precompiled analyzer on the Zeek command
line, you can also copy them into
${prefix}/lib/zeek/spicy
. Zeek will
automatically load any *.hlto
object files it finds there. In
addition, Zeek also scans its plugin directory for *.hlto
files. Alternatively, you can override both of those locations by
setting the environment variable ZEEK_SPICY_MODULE_PATH
to a set of
colon-separated directories to search instead. Zeek will then
only look there. In all cases, Zeek searches any directories
recursively, so it will find *.hlto
also if they are nested in
subfolders.
Run spicyz -h
to see some additional options it provides, which
are similar to https://docs.zeek.org/projects/spicy/en/latest/toolchain.html#spicy-driver.
Controlling Zeek from Spicy
Spicy grammars can import a provided library module zeek
to gain
access to Zeek-specific functions that call back into Zeek’s
processing:
function zeek::confirm_protocol()
[Deprecated] Triggers a DPD protocol confirmation for the current connection.
This function has been deprecated and will be removed. Use spicy::accept_input
instead, which will have the same effect with Zeek.
function zeek::reject_protocol(reason: string)
[Deprecated] Triggers a DPD protocol violation for the current connection.
This function has been deprecated and will be removed. Use spicy::decline_input
instead, which will have the same effect with Zeek.
function zeek::weird(id: string, addl: string = "") : &cxxname="zeek::spicy::rt::weird";
Reports a “weird” to Zeek. This should be used with similar semantics as in Zeek: something quite unexpected happening at the protocol level, which however does not prevent us from continuing to process the connection.
id: the name of the weird, which (just like in Zeek) should be a static
string identifying the situation reported (e.g., unexpected_command
).
addl: additional information to record along with the weird
function zeek::is_orig() : bool
Returns true if we’re currently parsing the originator side of a connection.
function zeek::uid() : string
Returns the current connection’s UID.
function zeek::conn_id() : tuple<orig_h: addr, orig_p: port, resp_h: addr, resp_p: port>
Returns the current connection’s 4-tuple ID to make IP address and port information available.
function zeek::flip_roles()
Instructs Zeek to flip the directionality of the current connection.
function zeek::number_packets() : uint64
Returns the number of packets seen so far on the current side of the current connection.
function zeek::protocol_begin(analyzer: optional<string>, protocol: spicy::Protocol = spicy::Protocol::TCP)
Adds a Zeek-side child protocol analyzer to the current connection.
If the same analyzer was added previously with protocol_handle_get_or_create or protocol_begin with same argument, and not closed with protocol_handle_close or protocol_end, no new analyzer will be added.
See protocol_handle_get_or_create for the error semantics of this function.
analyzer: type of analyzer to instantiate, specified through its Zeek-side name (similar to what Zeek’s signature action enable takes)
protocol: the transport-layer protocol that the analyzer uses; only TCP is currently supported here
Note: For backwards compatibility, the analyzer argument can be left unset to add a DPD analyzer. This use is deprecated, though; use the single-argument version of protocol_begin for that instead.
function zeek::protocol_begin(protocol: spicy::Protocol = spicy::Protocol::TCP)
Adds a Zeek-side DPD child protocol analyzer performing dynamic protocol detection on subsequently provided data.
If the same DPD analyzer was added previously with protocol_handle_get_or_create or protocol_begin with same argument, and not closed with protocol_handle_close or protocol_end, no new analyzer will be added.
See protocol_handle_get_or_create for the error semantics of this function.
protocol: the transport-layer protocol on which to perform protocol detection; only TCP is currently supported here
function zeek::protocol_handle_get_or_create(analyzer: string, protocol: spicy::Protocol = spicy::Protocol::TCP) : ProtocolHandle
Gets a handle to a Zeek-side child protocol analyzer for the current connection.
If no such child exists yet it will be added; otherwise a handle to the existing child protocol analyzer will be returned.
This function will return an error if:
not called from a protocol analyzer, or
the requested child protocol analyzer is of unknown type or not support by the requested transport protocol, or
creation of a child analyzer of the requested type was prevented by a previous call of disable_analyzer with prevent=T
analyzer: type of analyzer to get or instantiate, specified through its Zeek-side name (similar to what Zeek’s signature action enable takes).
protocol: the transport-layer protocol that the analyser uses; only TCP is currently supported here
function zeek::protocol_data_in(is_orig: bool, data: bytes, protocol: spicy::Protocol = spicy::Protocol::TCP)
Forwards protocol data to all previously instantiated Zeek-side child protocol analyzers of a given transport-layer.
is_orig: true to feed the data to the child’s originator side, false for the responder
data: chunk of data to forward to child analyzer
protocol: the transport-layer protocol of the children to forward to; only TCP is currently supported here
function zeek::protocol_data_in(is_orig: bool, data: bytes, h: ProtocolHandle)
Forwards protocol data to a specific previously instantiated Zeek-side child analyzer.
is_orig: true to feed the data to the child’s originator side, false for the responder
data: chunk of data to forward to child analyzer
h: handle to the child analyzer to forward data into
function zeek::protocol_gap(is_orig: bool, offset: uint64, len: uint64, h: optional<ProtocolHandle> = Null)
Signals a gap in input data to all previously instantiated Zeek-side child protocol analyzers.
is_orig: true to signal gap to the child’s originator side, false for the responder
offset: start offset of gap in input stream
len: size of gap
h: optional handle to the child analyzer signal a gap to, else signal to all child analyzers
function zeek::protocol_end()
Signals end-of-data to all previously instantiated Zeek-side child protocol analyzers and removes them.
function zeek::protocol_handle_close(handle: ProtocolHandle)
Signals end-of-data to the given child analyzer and removes it.
The given handle must be live, i.e., it must not have been used in a previous protocol_handle_close call, and must not have been live when protocol_end was called. If the handle is not live a runtime error will be triggered.
handle: handle to the child analyzer to remove
function zeek::file_begin(mime_type: optional<string> = Null, fuid: optional<string> = Null) : string
Signals the beginning of a file to Zeek’s file analysis, associating it with the current connection. Optionally, a mime type can be provided. It will be passed on to Zeek’s file analysis framework. Optionally, a file ID can be provided. It will be passed on to Zeek’s file analysis framework. Returns the Zeek-side file ID of the new file.
function zeek::fuid() : string
Returns the current file’s FUID.
function zeek::terminate_session()
Terminates the currently active Zeek-side session, flushing all state. Any subsequent activity will start a new session from scratch. This can only be called from inside a protocol analyzer.
function zeek::skip_input()
Tells Zeek to skip sending any further input data to the current analyzer. This is supported for protocol and file analyzers.
function zeek::file_set_size(size: uint64, fid: optional<string> = Null)
Signals the expected size of a file to Zeek’s file analysis.
size: expected size of file fid: Zeek-side ID of the file to operate on; if not given, the file started by the most recent file_begin() will be used
function zeek::file_data_in(data: bytes, fid: optional<string> = Null)
Passes file content on to Zeek’s file analysis.
data: chunk of raw data to pass into analysis fid: Zeek-side ID of the file to operate on; if not given, the file started by the most recent file_begin() will be used
function zeek::file_data_in_at_offset(data: bytes, offset: uint64, fid: optional<string> = Null)
Passes file content at a specific offset on to Zeek’s file analysis.
data: chunk of raw data to pass into analysis offset: position in file where data starts fid: Zeek-side ID of the file to operate on; if not given, the file started by the most recent file_begin() will be used
function zeek::file_gap(offset: uint64, len: uint64, fid: optional<string> = Null)
Signals a gap in a file to Zeek’s file analysis.
offset: position in file where gap starts len: size of gap fid: Zeek-side ID of the file to operate on; if not given, the file started by the most recent file_begin() will be used
function zeek::file_end(fid: optional<string> = Null)
Signals the end of a file to Zeek’s file analysis.
fid: Zeek-side ID of the file to operate on; if not given, the file started by the most recent file_begin() will be used
function zeek::forward_packet(identifier: uint32)
Inside a packet analyzer, forwards what data remains after parsing the top-level unit on to another analyzer. The index specifies the target, per the current dispatcher table.
function zeek::network_time() : time
Gets the network time from Zeek.
function zeek::get_address(id: string) : addr
Returns the value of a global Zeek script variable of Zeek type addr
.
Throws an exception if there’s no such Zeek of that name, or if it’s not of
the expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_bool(id: string) : bool
Returns the value of a global Zeek script variable of Zeek type bool
.
Throws an exception if there’s no such Zeek of that name, or if it’s not of
the expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_count(id: string) : uint64
Returns the value of a global Zeek script variable of Zeek type count
.
Throws an exception if there’s no such Zeek of that name, or if it’s not of
the expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_double(id: string) : real
Returns the value of a global Zeek script variable of Zeek type double
.
Throws an exception if there’s no such Zeek of that name, or if it’s not of
the expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_enum(id: string) : string
Returns the value of a global Zeek script variable of Zeek type enum
.
The value is returned as a string containing the enum’s label name, without
any scope. Throws an exception if there’s no such Zeek of that name, or if
it’s not of the expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_int(id: string) : int64
Returns the value of a global Zeek script variable of Zeek type int
.
Throws an exception if there’s no such Zeek of that name, or if it’s not of
the expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_interval(id: string) : interval
Returns the value of a global Zeek script variable of Zeek type
interval
. Throws an exception if there’s no such Zeek of that name, or
if it’s not of the expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_port(id: string) : port
Returns the value of a global Zeek script variable of Zeek type port
.
Throws an exception if there’s no such Zeek of that name, or if it’s not of
the expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_record(id: string) : ZeekRecord
Returns the value of a global Zeek script variable of Zeek type record
.
The value is returned as an opaque handle to the record, which can be used
with the zeek::record_*()
functions to access the record’s fields.
Throws an exception if there’s no such Zeek of that name, or if it’s not of
the expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_set(id: string) : ZeekSet
Returns the value of a global Zeek script variable of Zeek type set
. The
value is returned as an opaque handle to the set, which can be used with the
zeek::set_*()
functions to access the set’s content. Throws an exception
if there’s no such Zeek of that name, or if it’s not of the expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_string(id: string) : bytes
Returns the value of a global Zeek script variable of Zeek type string
.
The string’s value is returned as a Spicy bytes
value. Throws an
exception if there’s no such Zeek of that name, or if it’s not of the
expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_subnet(id: string) : network
Returns the value of a global Zeek script variable of Zeek type subnet
.
Throws an exception if there’s no such Zeek of that name, or if it’s not of
the expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_table(id: string) : ZeekTable
Returns the value of a global Zeek script variable of Zeek type table
.
The value is returned as an opaque handle to the set, which can be used with
the zeek::set_*()
functions to access the set’s content. Throws an
exception if there’s no such Zeek of that name, or if it’s not of the
expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_time(id: string) : time
Returns the value of a global Zeek script variable of Zeek type time
.
Throws an exception if there’s no such Zeek of that name, or if it’s not of
the expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_vector(id: string) : ZeekVector
Returns the value of a global Zeek script variable of Zeek type vector
.
The value is returned as an opaque handle to the vector, which can be used
with the zeek::vector_*()
functions to access the vector’s content.
Throws an exception if there’s no such Zeek of that name, or if it’s not of
the expected type.
id: fully-qualified name of the global Zeek variable to retrieve
function zeek::get_value(id: string) : ZeekVal
Returns an opaque handle to a global Zeek script variable. The handle can be
used with the zeek::as_*()
functions to access the variable’s value.
Throws an exception if there’s no Zeek variable of that name.
function zeek::as_address(v: ZeekVal) : addr
Returns a Zeek addr
value refereced by an opaque handle. Throws an
exception if the referenced value is not of the expected type.
function zeek::as_bool(v: ZeekVal) : bool
Returns a Zeek bool
value refereced by an opaque handle. Throws an
exception if the referenced value is not of the expected type.
function zeek::as_count(v: ZeekVal) : uint64
Returns a Zeek count
value refereced by an opaque handle. Throws an
exception if the referenced value is not of the expected type.
function zeek::as_double(v: ZeekVal) : real
Returns a Zeek double
value refereced by an opaque handle. Throws an
exception if the referenced value is not of the expected type.
function zeek::as_enum(v: ZeekVal) : string
Returns a Zeek enum
value refereced by an opaque handle. Throws an
exception if the referenced value is not of the expected type.
function zeek::as_int(v: ZeekVal) : int64
Returns a Zeek int
value refereced by an opaque handle. Throws an
exception if the referenced value is not of the expected type.
function zeek::as_interval(v: ZeekVal) : interval
Returns a Zeek interval
value refereced by an opaque handle. Throws an
exception if the referenced value is not of the expected type.
function zeek::as_port(v: ZeekVal) : port
Returns a Zeek port
value refereced by an opaque handle. Throws an
exception if the referenced value is not of the expected type.
function zeek::as_record(v: ZeekVal) : ZeekRecord
Returns a Zeek record
value refereced by an opaque handle. Throws an
exception if the referenced value is not of the expected type.
function zeek::as_set(v: ZeekVal) : ZeekSet
Returns a Zeek set
value refereced by an opaque handle. Throws an
exception if the referenced value is not of the expected type.
function zeek::as_string(v: ZeekVal) : bytes
Returns a Zeek string
value refereced by an opaque handle. The string’s
value is returned as a Spicy bytes
value. Throws an exception if the
referenced value is not of the expected type.
function zeek::as_subnet(v: ZeekVal) : network
Returns a Zeek subnet
value refereced by an opaque handle. Throws an
exception if the referenced value is not of the expected type.
function zeek::as_table(v: ZeekVal) : ZeekTable
Returns a Zeek table
value refereced by an opaque handle. Throws an
exception if the referenced value is not of the expected type.
function zeek::as_time(v: ZeekVal) : time
Returns a Zeek time
value refereced by an opaque handle. Throws an
exception if the referenced value is not of the expected type.
function zeek::as_vector(v: ZeekVal) : ZeekVector
Returns a Zeek vector
value refereced by an opaque handle. Throws an
exception if the referenced value is not of the expected type.
function zeek::set_contains(id: string, v: any) : bool
Returns true if a Zeek set contains a given value. Throws an exception if the given ID does not exist, or does not have the expected type.
id: fully-qualified name of the global Zeek set to check v: value to check for, which must be of the Spicy-side equivalent of the set’s key type
function zeek::set_contains(s: ZeekSet, v: any) : bool
Returns true if a Zeek set contains a given value. Throws an exception if the set does not have the expected type.
s: opaque handle to the Zeek set, as returned by other functions v: value to check for, which must be of the Spicy-side equivalent of the set’s key type
function zeek::table_contains(id: string, v: any) : bool
Returns true if a Zeek table contains a given value. Throws an exception if the given ID does not exist, or does not have the expected type.
id: fully-qualified name of the global Zeek table to check v: value to check for, which must be of the Spicy-side equivalent of the table’s key type
function zeek::table_contains(t: ZeekTable, v: any) : bool
Returns true if a Zeek table contains a given value. Throws an exception if the given ID does not exist, or does not have the expected type.
t: opaque handle to the Zeek table, as returned by other functions v: value to check for, which must be of the Spicy-side equivalent of the table’s key type
function zeek::table_lookup(id: string, v: any) : optional<ZeekVal>
Returns the value associated with a key in a Zeek table. Returns an error result if the key does not exist in the table. Throws an exception if the given table ID does not exist, or does not have the expected type.
id: fully-qualified name of the global Zeek table to check v: value to lookup, which must be of the Spicy-side equivalent of the table’s key type
function zeek::table_lookup(t: ZeekTable, v: any) : optional<ZeekVal>
Returns the value associated with a key in a Zeek table. Returns an error result if the key does not exist in the table. Throws an exception if the given table ID does not exist, or does not have the expected type.
t: opaque handle to the Zeek table, as returned by other functions v: value to lookup, which must be of the Spicy-side equivalent of the table’s key type
function zeek::record_has_value(id: string, field: string) : bool
Returns true if a Zeek record provides a value for a given field. This includes fields with &default values. Throws an exception if the given ID does not exist, or does not have the expected type.
id: fully-qualified name of the global Zeek record to check field: name of the field to check
function zeek::record_has_value(r: ZeekRecord, field: string) : bool
Returns true if a Zeek record provides a value for a given field. This includes fields with &default values.
r: opaque handle to the Zeek record, as returned by other functions field: name of the field to check
function zeek::record_has_field(id: string, field: string) : bool
Returns true if the type of a Zeek record has a field of a given name. Throws an exception if the given ID does not exist, or does not have the expected type.
id: fully-qualified name of the global Zeek record to check field: name of the field to check
function zeek::record_has_field(r: ZeekRecord, field: string) : bool
Returns true if the type of a Zeek record has a field of a given name.
r: opaque handle to the Zeek record, as returned by other functions field: name of the field to check
function zeek::record_field(id: string, field: string) : ZeekVal
Returns a field’s value from a Zeek record. Throws an exception if the given ID does not exist, or does not have the expected type; or if there’s no such field in the record type, or if the field does not have a value.
id: fully-qualified name of the global Zeek record to check field: name of the field to retrieve
function zeek::record_field(r: ZeekRecord, field: string) : ZeekVal
Returns a field’s value from a Zeek record. Throws an exception if the given record does not have such a field, or if the field does not have a value.
r: opaque handle to the Zeek record, as returned by other functions field: name of the field to retrieve
function zeek::vector_index(id: string, index: uint64) : ZeekVal
Returns the value of an index in a Zeek vector. Throws an exception if the given ID does not exist, or does not have the expected type; or if the index is out of bounds.
id: fully-qualified name of the global Zeek vector to check index: index of the element to retrieve
function zeek::vector_index(v: ZeekVector, index: uint64) : ZeekVal
Returns the value of an index in a Zeek vector. Throws an exception if the index is out of bounds.
v: opaque handle to the Zeek vector, as returned by other functions index: index of the element to retrieve
function zeek::vector_size(id: string) : uint64
Returns the size of a Zeek vector. Throws an exception if the given ID does not exist, or does not have the expected type.
id: fully-qualified name of the global Zeek vector to check
function zeek::vector_size(v: ZeekVector) : uint64
Returns the size of a Zeek vector.
v: opaque handle to the Zeek vector, as returned by other functions
… zeek_variables:
Accessing Zeek Variables from Spicy
You can access Zeek-side global variables from inside Spicy, which is
particularly handy for configuring Spicy analyzers from Zeek script
code. The zeek module facilitates this
through a set of functions coverting the current value of Zeek
variables into corresponding Spicy values. For example, let’s say you
would like to provide a Zeek script option with a count
value that
your Spicy analyzer can leverage. On the Zeek side, you’d define the
option like this:
module MyModule;
export {
option my_value: count &default=42;
}
Then, in your Spicy code, you can access the value of that option like this:
import zeek;
...
local my_value = zeek::get_count("MyModule::my_value");
...
This looks up the option by its global, fully-scoped ID and returns the
Zeek-side count
as a Spicy uint64
. If there are any errors,
such as the global not existing or having the wrong type, the
get_count()
would abort with an exception.
There are corresponding conversion functions for most of Zeek’s
built-in types, see Controlling Zeek from Spicy for the full list of
get_<TYPE>()
functions.
For Zeek container types (map
, record
, set
, vector
),
the get_<TYPE>()
functions return an opaque value that can be used
to then inspect the container further. For example, you can check
membership in a set
like this:
# Zeek module `MyModule`
option my_set: set[count] = { 1, 2, 3 };
# Spicy
local my_set = zeek::get_set("MyModule::my_set");
if ( zeek::set_contains(my_set, 2) )
print "Found 2 in the set";
For retrieving values from containers, there are further as_<TYPE>
functions for conversion. Example accessing a record’s field:
# Zeek module `MyModule`
option my_record: record {
a: count &default = 42;
b: string &default = "foo";
};
# Spicy
local my_record = zeek::get_record("MyModule::my_record");
local a = zeek::as_count(zeek::record_field(my_record, "a"));
local b = zeek::as_string(zeek::record_field(my_record, "b")); # returns "bytes" (not string)
See Controlling Zeek from Spicy again for the full list of as_<TYPE>()
and container accessor functions.
The API provides only read access to Zeek variables, there’s no way to change them from inside Spicy. This is both to simplify the API, and also conceptually to avoid offering a back channel into Zeek state that could end up producing a very tight coupling of Spicy and Zeek code. Use events to communicate from Spicy to Zeek instead.
Access to a Zeek global can be relatively slow as it performs the ID
lookup and data conversion. Accordingly, the mechanism is primarily
meant for configuration-style data, not for transferring heaps of
dynamic state. However, as a way to speed up repeated access slightly,
you can use zeek::get_value to cache the
result of the ID lookup, then use as_<TYPE>()
on the cached value
to retrieve the actual value.
Note
When accessing global Zeek variables from Spicy, take into account
whether their Zeek values might change over time. If you’re
accessing a Zeek const
, you can be sure it won’t change, so
you could just store its value inside a corresponding Spicy
global
at initialization time, and hence avoid repeated
lookups during runtime. However, if you’re accessing a Zeek
option
, those are designed to support runtime updates, so you
should not cache their values on the Spicy side. Likewise, any
Zeek global
can of course change anytime. (In both cases, you
can still use get_value()
to cache the ID lookup.)
Dynamic Protocol Detection (DPD)
Spicy protocol analyzers support Zeek’s Dynamic Protocol Detection (DPD), i.e., analysis independent of any well-known ports. To use that with your analyzer, add two pieces:
A Zeek signature to activate your analyzer based on payload patterns. Just like with any of Zeek’s standard analyzers, a signature can activate a Spicy analyzer through the
enable "<name>"
keyword. The name of the analyzer comes out of the EVT file: it is theANALYZER_NAME
with the double colons replaced with an underscore (e.g.,spicy::HTTP
turns intoenable "spicy_HTTP"
.You should call spicy::accept_input() from a hook inside your grammar at a point when the parser can be reasonably certain that it is processing the expected protocol. Optionally, you may also call spicy::decline_input() when you’re sure the parser is not parsing the right protocol. However, Zeek will also trigger this automatically whenever your parser aborts with an error.
Configuration
Options
Zeek provides a set of script-level options to tune Spicy
behavior. These all live in the Spicy::
namespace:
## Constant for testing if Spicy is available.
const available = T;
## Show output of Spicy print statements.
const enable_print = F &redef;
# Record and display profiling information, if compiled into analyzer.
const enable_profiling = F &redef;
## abort() instead of throwing HILTI exceptions.
const abort_on_exceptions = F &redef;
## Include backtraces when reporting unhandled exceptions.
const show_backtraces = F &redef;
## Maximum depth of recursive file analysis (Spicy analyzers only)
const max_file_depth: count = 5 &redef;
Functions
Zeek also adds the following new built-in functions for Spicy, which
likewise live in the Spicy::
namespace:
## Enable a specific Spicy protocol analyzer if not already active. If this
## analyzer replaces an standard analyzer, that one will automatically be
## disabled.
##
## tag: analyzer to toggle
##
## Returns: true if the operation succeeded
global enable_protocol_analyzer: function(tag: Analyzer::Tag) : bool;
## Disable a specific Spicy protocol analyzer if not already inactive. If
## this analyzer replaces an standard analyzer, that one will automatically
## be re-enabled.
##
## tag: analyzer to toggle
##
## Returns: true if the operation succeeded
global disable_protocol_analyzer: function(tag: Analyzer::Tag) : bool;
## Enable a specific Spicy file analyzer if not already active. If this
## analyzer replaces an standard analyzer, that one will automatically be
## disabled.
##
## tag: analyzer to toggle
##
## Returns: true if the operation succeeded
global enable_file_analyzer: function(tag: Files::Tag) : bool;
## Disable a specific Spicy file analyzer if not already inactive. If
## this analyzer replaces an standard analyzer, that one will automatically
## be re-enabled.
##
## tag: analyzer to toggle
##
## Returns: true if the operation succeeded
global disable_file_analyzer: function(tag: Files::Tag) : bool;
## Returns current resource usage as reported by the Spicy runtime system.
global resource_usage: function() : ResourceUsage;
Debugging
If Zeek doesn’t seem to be doing the right thing with your Spicy
analyzer, there are several ways to debug what’s going on. To
facilitate that, compile your analyzer with spicyz -d
and, if
possible, use a debug version of Zeek (i.e., build Zeek with
./configure --enable-debug
).
If your analyzer doesn’t seem to be active at all, first make sure
Zeek actually knows about it: It should show up in the output of
zeek -NN Zeek::Spicy
. If it doesn’t, you might not have been using
the right *.spicy
or *.evt
files when precompiling, or Zeek is
not loading the *.hlto
file. Also check your *.evt
if it
defines your analyzer correctly.
If Zeek knows about your analyzer and just doesn’t seem to activate
it, double-check that ports or MIME types are correct in the *.evt
file. If you’re using a signature instead, try a port/MIME type first,
just to make sure it’s not a matter of signature mismatches.
If there’s nothing obviously wrong with your source files, you can
trace what the Zeek’s Spicy support is compiling by running spicyz
with -D zeek
. For example, reusing the HTTP example from the Getting Started guide:
# spicyz -D zeek my-http.spicy my-http.evt -o my-http.hlt
[debug/zeek] Loading Spicy file "/Users/robin/work/spicy/main/tests/spicy/doc/my-http.spicy"
[debug/zeek] Loading EVT file "/Users/robin/work/spicy/main/doc/examples/my-http.evt"
[debug/zeek] Loading events from /Users/robin/work/spicy/main/doc/examples/my-http.evt
[debug/zeek] Got protocol analyzer definition for spicy_MyHTTP
[debug/zeek] Got event definition for MyHTTP::request_line
[debug/zeek] Running Spicy driver
[debug/zeek] Got unit type 'MyHTTP::Version'
[debug/zeek] Got unit type 'MyHTTP::RequestLine'
[debug/zeek] Adding protocol analyzer 'spicy_MyHTTP'
[debug/zeek] Adding Spicy hook 'MyHTTP::RequestLine::0x25_done' for event MyHTTP::request_line
[debug/zeek] Done with Spicy driver
You can see the main pieces in there: The files being loaded, unit types provided by them, analyzers and events being created.
If that all looks as expected, it’s time to turn to the Zeek side and
see what it’s doing at runtime. You’ll need a debug version of Zeek
for that, as well as a small trace with traffic that you expect your
analyzer to process. Run Zeek with -B dpd
(or -B file_analysis
if you’re debugging a file analyzer) on your trace to record the
analyzer activity into debug.log
. For example, with the same HTTP
example, we get:
1# zeek -B dpd -Cr request-line.pcap my-http.hlto
2# cat debug.log
3[dpd] Registering analyzer SPICY_MYHTTP for port 12345/1
4[...[
5[dpd] Available analyzers after spicy_init():
6[...]
7[dpd] spicy_MyHTTP (enabled)
8[...]
9[dpd] Analyzers by port:
10[dpd] 12345/tcp: SPICY_MYHTTP
11[...]
12[dpd] TCP[5] added child SPICY_MYHTTP[7]
13[dpd] 127.0.0.1:59619 > 127.0.0.1:12345 activated SPICY_MYHTTP analyzer due to port 12345
14[...]
15[dpd] SPICY_MYHTTP[7] DeliverStream(25, T) [GET /index.html HTTP/1.0\x0a]
16[dpd] SPICY_MYHTTP[7] EndOfData(T)
17[dpd] SPICY_MYHTTP[7] EndOfData(F)
The first few lines show that Zeek’s analyzer system registers the analyzer as expected. The subsequent lines show that the analyzer gets activated for processing the connection in the trace, and that it then receives the data that we know indeed constitutes its payload, before it eventually gets shutdown.
To see this from the Zeek side, set the zeek
debug stream
through the HILTI_DEBUG
environment variable:
# HILTI_DEBUG=zeek zeek -Cr request-line.pcap my-http.hlto
[zeek] Have Spicy protocol analyzer spicy_MyHTTP
[zeek] Registering Protocol::TCP protocol analyzer spicy_MyHTTP with Zeek
[zeek] Scheduling analyzer for port 12345/tcp
[zeek] Done with post-script initialization
[zeek] [SPICY_MYHTTP/7/orig] initial chunk: |GET /index.html HTTP/1.0\\x0a| (eod=false)
[zeek] [SPICY_MYHTTP/7/orig] -> event MyHTTP::request_line($conn, GET, /index.html, 1.0)
[zeek] [SPICY_MYHTTP/7/orig] done with parsing
[zeek] [SPICY_MYHTTP/7/orig] parsing finished, skipping further originator payload
[zeek] [SPICY_MYHTTP/7/resp] no unit specified for parsing
[zeek] [SPICY_MYHTTP/7/orig] skipping end-of-data delivery
[zeek] [SPICY_MYHTTP/7/resp] no unit specified for parsing
[zeek] [SPICY_MYHTTP/7/orig] skipping end-of-data delivery
[zeek] [SPICY_MYHTTP/7/resp] no unit specified for parsing
After the initial initialization, you see the data arriving and the event being generated for Zeek. Zeek also reports that we didn’t define a unit for the responder side—which we know in this case, but if that appears unexpectedly you probably found a problem.
So we know now that our analyzer is receiving the anticipated data to
parse. At this point, we can switch to debugging the Spicy side
through the usual mechanisms. In particular,
setting HILTI_DEBUG=spicy
tends to be helpful:
# HILTI_DEBUG=spicy zeek -Cr request-line.pcap my-http.hlto
[spicy] MyHTTP::RequestLine
[spicy] method = GET
[spicy] anon_2 =
[spicy] uri = /index.html
[spicy] anon_3 =
[spicy] MyHTTP::Version
[spicy] anon = HTTP/
[spicy] number = 1.0
[spicy] version = [$number=b"1.0"]
[spicy] anon_4 = \n
If everything looks right with the parsing, and the right events are
generated too, then the final part is to check out the events that
arrive on the Zeek side. To get Zeek to see an event that Zeek
raises, you need to have at least one handler implemented for it in
one of your Zeek scripts. You can then load Zeek’s
misc/dump-events
to see them as they are being received, including
their full Zeek-side values:
# zeek -Cr request-line.pcap my-http.hlto misc/dump-events
[...]
1580991211.780489 MyHTTP::request_line
[0] c: connection = [id=[orig_h=127.0.0.1, orig_p=59619/tcp, ...] ...]
[1] method: string = GET
[2] uri: string = /index.html
[3] version: string = 1.0
[...]