known-services.zeek

Known

This script logs and tracks active services. For this script, an active service is defined as an IP address and port of a server for which a TCP handshake (SYN+ACK) is observed, assumed to have been done in the past (started seeing packets mid-connection, but the server is actively sending data), or sent at least one UDP packet. If a protocol name is found/known for service, that will be logged, but services whose names can’t be determined are also still logged.

Namespace:: Known
Imports:: base/frameworks/cluster, base/frameworks/storage/async.zeek, base/frameworks/storage/sync.zeek, base/utils/directions-and-hosts.zeek, policy/frameworks/storage/backend/sqlite

Summary

Runtime Options

`Known::service_store_timeout`: `interval` `&redef`	The timeout interval to use for operations against `Known::service_broker_store` and `Known::service_store_backend`.
`Known::service_tracking`: `Host` `&redef`	The hosts whose services should be tracked and logged.
`Known::service_udp_requires_response`: `bool` `&redef`	Require UDP server to respond before considering it an “active service”.

Redefinable Options

`Known::service_store_backend_options`: `Storage::BackendOptions` `&redef`	The options for the service store.
`Known::service_store_backend_type`: `Storage::Backend` `&redef`	The type of storage backend to open.
`Known::service_store_expiry`: `interval` `&redef`	The expiry interval of new entries in `Known::service_broker_store` and `Known::service_store_backend`.
`Known::service_store_name`: `string` `&redef`	The Broker topic name to use for `Known::service_broker_store`.
`Known::service_store_prefix`: `string` `&redef`	The name to use for `Known::service_store_backend`.
`Known::use_service_store`: `bool` `&redef`	Toggles between different implementations of this script.
`Known::use_storage_framework`: `bool` `&redef`	Switches to the version of this script that uses the storage framework instead of Broker stores.

State Variables

`Known::service_broker_store`: `Cluster::StoreInfo`	Storage configuration for Broker stores Holds the set of all known services.
`Known::service_store_backend`: `opaque`	Storage configuration for storage framework stores This requires setting a configuration in local.zeek that sets the Known::use_storage_framework boolean to T, and optionally sets different values in the Known::service_store_backend_options record.
`Known::services`: `table` `&create_expire` = `1.0 day`	Tracks the set of daily-detected services for preventing the logging of duplicates, but can also be inspected by other scripts for different purposes.

Types

`Known::AddrPortServTriplet`: `record`
`Known::ServicesInfo`: `record`	The record type which contains the column fields of the known-services log.

Redefinitions

Log::ID: enum

The known-services logging stream identifier.

Known::SERVICES_LOG

connection: record

New Fields:

connection

known_services_done: bool &default = F &optional

Events

Known::log_known_services: event

Event that can be handled to access the Known::ServicesInfo record as it is sent on to the logging framework.

Hooks

Known::log_policy_services: Log::PolicyHook

A default logging policy hook for the stream.

Detailed Interface

Runtime Options

Known::service_store_timeout 

Type:: interval
Attributes:: &redef
Default:: 15.0 secs

The timeout interval to use for operations against Known::service_broker_store and Known::service_store_backend.

Known::service_tracking 

Type:

Host

Attributes:

&redef

Default:

LOCAL_HOSTS

Redefinition:

from policy/tuning/track-all-assets.zeek

=:

``ALL_HOSTS``

The hosts whose services should be tracked and logged. See Host for possible choices.

Known::service_udp_requires_response 

Type:: bool
Attributes:: &redef
Default:: T

Require UDP server to respond before considering it an “active service”.

Redefinable Options

Known::service_store_backend_options 

Type:

Storage::BackendOptions

Attributes:

&redef

Default:

{
   serializer=Storage::STORAGE_SERIALIZER_JSON
   forced_sync=F
   redis=<uninitialized>
   sqlite=[database_path=":memory:", table_name="zeek/known/services", busy_timeout=5.0 secs, pragma_commands={
      ["integrity_check"] = "",
      ["journal_mode"] = "WAL",
      ["synchronous"] = "normal",
      ["temp_store"] = "memory"
   }, pragma_timeout=500.0 msecs, pragma_wait_on_busy=5.0 msecs]
}

The options for the service store. This should be redef’d in local.zeek to set connection information for the backend. The options default to a memory store.

Known::service_store_backend_type 

Type:: Storage::Backend
Attributes:: &redef
Default:: Storage::STORAGE_BACKEND_SQLITE

The type of storage backend to open.

Known::service_store_expiry 

Type:: interval
Attributes:: &redef
Default:: 1.0 day

The expiry interval of new entries in Known::service_broker_store and Known::service_store_backend. This also changes the interval at which services get logged.

Known::service_store_name 

Type:: string
Attributes:: &redef
Default:: "zeek/known/services"

The Broker topic name to use for Known::service_broker_store.

Known::service_store_prefix 

Type:: string
Attributes:: &redef
Default:: "zeekknownservices"

The name to use for Known::service_store_backend. This will be used by the backends to differentiate tables/keys. This should be alphanumeric so that it can be used as the table name for the storage framework.

Known::use_service_store 

Type:: bool
Attributes:: &redef
Default:: F

Toggles between different implementations of this script. When true, use a Broker data store, else use a regular Zeek set with keys uniformly distributed over proxy nodes in cluster operation.

Known::use_storage_framework 

Type:: bool
Attributes:: &redef
Default:: F

Switches to the version of this script that uses the storage framework instead of Broker stores. This will default to T in v8.1.

State Variables

Known::service_broker_store 

Type:

Cluster::StoreInfo

Default:

{
   name=<uninitialized>
   store=<uninitialized>
   master_node=""
   master=F
   backend=Broker::MEMORY
   options=[sqlite=[path="", synchronous=<uninitialized>, journal_mode=<uninitialized>, failure_mode=Broker::SQLITE_FAILURE_MODE_FAIL, integrity_check=F]]
   clone_resync_interval=10.0 secs
   clone_stale_interval=5.0 mins
   clone_mutation_buffer_interval=2.0 mins
}

Storage configuration for Broker stores Holds the set of all known services. Keys in the store are Known::AddrPortServTriplet and their associated value is always the boolean value of “true”.

Known::service_store_backend 

Type:: opaque of Storage::BackendHandle

Storage configuration for storage framework stores This requires setting a configuration in local.zeek that sets the Known::use_storage_framework boolean to T, and optionally sets different values in the Known::service_store_backend_options record. Backend to use for storing known services data using the storage framework.

Known::services 

Type:: table [addr, port] of set [string]
Attributes:: &create_expire = 1.0 day
Default:: {}

Tracks the set of daily-detected services for preventing the logging of duplicates, but can also be inspected by other scripts for different purposes.

In cluster operation, this table is uniformly distributed across proxy nodes.

This table is automatically populated and shouldn’t be directly modified.