urls.zeek

Functions for URL handling.

Summary

Redefinable Options

url_regex: pattern &redef

A regular expression for matching and extracting URLs.

Types

URI: record

A URI, as parsed by decompose_uri.

Functions

`decompose_uri`: `function`
`find_all_urls`: `function`	Extracts URLs discovered in arbitrary text.
`find_all_urls_without_scheme`: `function`	Extracts URLs discovered in arbitrary text without the URL scheme included.

Detailed Interface

Redefinable Options

url_regex 

Type

pattern

Attributes

&redef

Default

/^?(^([a-zA-Z\-]{3,5}):\/\/(-\.)?([^[:blank:]\/?\.#-]+\.?)+(\/[^[:blank:]]*)?)$?/

A regular expression for matching and extracting URLs. This is the @imme_emosol regex from https://mathiasbynens.be/demo/url-regex, adapted for Zeek. It’s not perfect for all of their test cases, but it’s one of the shorter ones that covers most of the test cases.

Types

URI 

Type

record

scheme: string &optional: The URL’s scheme..
netlocation: string: The location, which could be a domain name or an IP address. Left empty if not specified.
portnum: count &optional: Port number, if included in URI.
path: string: Full including the file name. Will be ‘/’ if there’s not path given.
file_name: string &optional: Full file name, including extension, if there is a file name.
file_base: string &optional: The base filename, without extension, if there is a file name.
file_ext: string &optional: The filename’s extension, if there is a file name.
params: table [string] of string &optional: A table of all query parameters, mapping their keys to values, if there’s a query.

A URI, as parsed by decompose_uri.

Functions

decompose_uri 

Type: function (uri: string) : URI

find_all_urls 

Type: function (s: string) : string_set

Extracts URLs discovered in arbitrary text.

find_all_urls_without_scheme 

Type: function (s: string) : string_set

Extracts URLs discovered in arbitrary text without the URL scheme included.