known-hosts.zeek

Known

This script logs hosts that Zeek determines have performed complete TCP handshakes and logs the address once per day (by default). The log that is output provides an easy way to determine a count of the IP addresses in use on a network per day.

Namespace:: Known
Imports:: base/frameworks/cluster, base/frameworks/storage/async.zeek, base/frameworks/storage/sync.zeek, base/utils/directions-and-hosts.zeek, policy/frameworks/storage/backend/sqlite

Summary

Runtime Options

`Known::host_store_timeout`: `interval` `&redef`	The timeout interval to use for operations against `Known::host_broker_store` and `Known::host_store_backend`.
`Known::host_tracking`: `Host` `&redef`	The hosts whose existence should be logged and tracked.

Redefinable Options

`Known::enable_hosts_persistence`: `bool` `&redef`	Use the storage framework to enable persistence of the stored hosts between runs.
`Known::host_store_backend_options`: `Storage::BackendOptions` `&redef`	The options for the host store.
`Known::host_store_backend_type`: `Storage::Backend` `&redef`	The type of storage backend to open.
`Known::host_store_expiry`: `interval` `&redef`	The expiry interval of new entries in `Known::host_broker_store` and `Known::host_store_backend`.
`Known::host_store_name`: `string` `&redef`	The Broker topic name to use for `Known::host_broker_store`.
`Known::host_store_prefix`: `string` `&redef`	The name to use for `Known::host_store_backend`.
`Known::use_host_store`: `bool` `&redef` `&deprecated` = …	Toggles between different implementations of this script.

State Variables

`Known::host_broker_store`: `Cluster::StoreInfo`	Holds the set of all known hosts.
`Known::host_store_backend`: `opaque`	This requires setting a configuration in local.zeek that sets the Known::enable_hosts_persistence boolean to T, and optionally setting different values in the Known::host_store_backend_options record.
`Known::hosts`: `set` `&create_expire` = `1.0 day` `&redef`	The set of all known addresses to store for preventing duplicate logging of addresses.

Types

Known::HostsInfo: record

The record type which contains the column fields of the known-hosts log.

Redefinitions

Log::ID: enum

The known-hosts logging stream identifier.

Known::HOSTS_LOG

Events

Known::log_known_hosts: event

An event that can be handled to access the Known::HostsInfo record as it is sent on to the logging framework.

Hooks

Known::log_policy_hosts: Log::PolicyHook

A default logging policy hook for the stream.

Detailed Interface

Runtime Options

Known::host_store_timeout 

Type:: interval
Attributes:: &redef
Default:: 15.0 secs

The timeout interval to use for operations against Known::host_broker_store and Known::host_store_backend.

Known::host_tracking 

Type:

Host

Attributes:

&redef

Default:

LOCAL_HOSTS

Redefinition:

from policy/tuning/track-all-assets.zeek

=:

``ALL_HOSTS``

The hosts whose existence should be logged and tracked. See Host for possible choices.

Redefinable Options

Known::enable_hosts_persistence 

Type:: bool
Attributes:: &redef
Default:: F

Use the storage framework to enable persistence of the stored hosts between runs.

Known::host_store_backend_options 

Type:

Storage::BackendOptions

Attributes:

&redef

Default:

{
   serializer=Storage::STORAGE_SERIALIZER_JSON
   forced_sync=F
   redis=<uninitialized>
   sqlite=[database_path="/known/hosts.sqlite", table_name="zeekknownhosts", busy_timeout=5.0 secs, pragma_commands={
      ["quick_check"] = "",
      ["journal_mode"] = "WAL",
      ["synchronous"] = "normal",
      ["temp_store"] = "memory"
   }, pragma_timeout=500.0 msecs, pragma_wait_on_busy=5.0 msecs]
}

The options for the host store. This should be redef’d in local.zeek to set connection information for the backend. The options default to a central persistent sqlite database.

Known::host_store_backend_type 

Type:: Storage::Backend
Attributes:: &redef
Default:: Storage::STORAGE_BACKEND_SQLITE

The type of storage backend to open.

Known::host_store_expiry 

Type:: interval
Attributes:: &redef
Default:: 1.0 day

The expiry interval of new entries in Known::host_broker_store and Known::host_store_backend. This also changes the interval at which hosts get logged.

Known::host_store_name 

Type:: string
Attributes:: &redef
Default:: "zeek/known/hosts"

The Broker topic name to use for Known::host_broker_store.

Known::host_store_prefix 

Type:: string
Attributes:: &redef
Default:: "zeekknownhosts"

The name to use for Known::host_store_backend. This will be used by the backends to differentiate tables/keys. This should be alphanumeric so that it can be used as the table name for the storage framework.

Known::use_host_store 

Type:: bool
Attributes:: &redef &deprecated = “Remove in v9.1. Store support has been disabled by default since Zeek 6.0 due to performance and will be removed.”
Default:: F

Toggles between different implementations of this script. When true, use a Broker data store, else use a regular Zeek set with keys uniformly distributed over proxy nodes in cluster operation.

State Variables

Known::host_broker_store 

Type:

Cluster::StoreInfo

Default:

{
   name=<uninitialized>
   store=<uninitialized>
   master_node=""
   master=F
   backend=Broker::MEMORY
   options=[sqlite=[path="", synchronous=<uninitialized>, journal_mode=<uninitialized>, failure_mode=Broker::SQLITE_FAILURE_MODE_FAIL, integrity_check=F]]
   clone_resync_interval=10.0 secs
   clone_stale_interval=5.0 mins
   clone_mutation_buffer_interval=2.0 mins
}

Holds the set of all known hosts. Keys in the store are addresses and their associated value will always be the “true” boolean.

Known::host_store_backend 

Type:: opaque of Storage::BackendHandle

This requires setting a configuration in local.zeek that sets the Known::enable_hosts_persistence boolean to T, and optionally setting different values in the Known::host_store_backend_options record. Backend to use for storing known hosts data using the storage framework.

Known::hosts 

Type:: set [addr]
Attributes:: &create_expire = 1.0 day &redef
Default:: {}

The set of all known addresses to store for preventing duplicate logging of addresses. It can also be used from other scripts to inspect if an address has been seen in use. Maintain the list of known hosts for 24 hours so that the existence of each individual address is logged each day.

In cluster operation, this set is distributed uniformly across proxy nodes.