GeoIP and DNS Enrichment in Cribl Stream

GeoIP

Cribl Stream's GeoIP Function enriches events with geographic fields, given an IP address, per the GeoIP Function docs. It works with the binary database from MaxMind's GeoIP or from IPinfo's IP address download.

The prerequisite is that you host the .mmdb database file yourself. The page's steps are to download and extract the file, then determine where to place it, with the $CRIBL_HOME/state/ subdirectory recommended, already listed in the default .gitignore file. You can also upload it to Cribl Stream's Lookups Library, and in a Cribl.Cloud deployment that is currently the only option. Place the file on your Worker Node(s): in a distributed deployment the page recommends the Leader and all Worker Nodes, while smaller deployments can get away with the Leader Node only. Optionally, set up automatic updates to the MaxMind and IPinfo database files.

The page documents these settings, shown with their default values:

• Filter: Filter expression (JavaScript) that selects data to feed through the Function. Defaults to true, meaning it evaluates all events.
• Description: Simple description about this Function. Defaults to empty.
• Final: Toggle on to stop feeding data to the downstream Functions. Default is toggled off.
• GeoIP file (.mmdb): Path to the database, in binary format, with .mmdb extension. If the database file is located within the lookup directory ($CRIBL_HOME/data/lookups/), it does not need to be an absolute path. In distributed deployments, the file must be in the same location on the Leader Node and all Worker Nodes.
• IP field: Field name in which to find an IP to look up. It can be nested. Defaults to ip.
• Result field: Field name in which to store the GeoIP lookup results. Defaults to geoip.

The page's example works from SMTP logs and chains three Functions: a Regex Extract Function isolates the host's IP, the GeoIP Function looks it up and places the returned info into a new __geoip field, and an Eval Function parses that field. The first two steps, copied from the GeoIP Function docs:

Function 1 - Regex Extract
Regex: \[(?<ip>\S+)\]
Source field: _raw
Result: 74.136.132.88

Function 2 - GeoIP
Event's IP field: ip
Result field: __geoip

The Eval step turns the stored result into readable values. Its name and value-expression pairs, copied from the GeoIP Function docs:

City    __geoip.city.names.en
Country __geoip.country.names.en
Zip     __geoip.postal.code
Lat     __geoip.location.latitude
Long    __geoip.location.longitude

The page adds that you could specify the __geoip field in the Eval Function's Remove fields setting, if desired. However, its __ prefix makes it an internal field anyway. It also points to Cribl's GeoIP and Threat Feed Enrichment Sandbox for a hosted tutorial.

DNS lookup

The DNS Lookup Function offers two operations useful in enriching security and other data, per the DNS Lookup Function docs. The first resolves a host name as text to an A record (an IP address) or to other record types. The second, a reverse DNS lookup, duplicates the behavior of Cribl Stream's prior Reverse DNS Function, which was deprecated and has since been removed.

To reduce DNS lookups and minimize latency, the Function incorporates a configurable DNS cache covering both resolved and unresolved lookups. For additional caching, the page suggests OS-level DNS caching on each Worker that executes this Function, naming DNSMasq, nscd, and systemd-resolved.

The page documents these settings:

• Filter, Description, and Final match the GeoIP Function: Filter defaults to true, meaning it evaluates all events; Description defaults to empty; and Final is toggled off by default.
• DNS Lookup Fields section: Lookup field name names the field containing the domain to look up; Resource record type sets the DNS record type (RR) to return, defaulting to the A record; and Output field name names the field the lookup result(s) are added to, where leaving it blank overwrites the original field specified in Lookup field name.
• Reverse DNS Lookup section: Lookup field name names the field containing the IP address to look up, and the page flags in a warning note that if the field value is not in IPv4 or IPv6 format, the lookup is skipped. Output field name names the field in which to add the resolved hostname, and leaving it blank overwrites the original field specified in Lookup field name.
• Fallback Resolving Methods section, for resolving DNS short names: Use /etc/resolv.conf (default is toggled off, not applicable for Windows hosts), Use search or domain fallback(s) (fallback value(s) for the resolver when it cannot resolve a short name), and Fall back to DNS.lookup(). If you enable all of these methods at once, Cribl Stream tries them in this order: Use /etc/resolv.conf, then the search or domain fallback(s), then DNS.lookup(). If the short name still cannot be resolved after all of these methods, it will drop the output field from the event.
• Advanced Settings: DNS server overrides (IP address(es) in RFC 5952 format of the DNS server(s) to use for resolution. If the field is not specified, Cribl Stream uses the system's DNS server), Cache time to live (minutes) (defaults to 30 minutes. Use 0 to disable cache expiration/refresh behavior), Cache size limit (default 5000. Higher values increase memory usage. Highest allowed value is 100000), and Log level for failed lookups (defaults to error).

The page describes monitoring cache state: a hit is a lookup served from the cache, which skips contacting a DNS server, and a miss forces that contact. Stats log to the func:dns_lookup channel at the debug level and appear after every 5,000 cache lookups, with fields hits, misses, keys, ksize (approximate bytes in keys), and vsize (approximate bytes in values).

The page's example Pipeline chains two Functions: an Eval Function defines key-value pairs for two domain names and two IP addresses, then the DNS Lookup Function looks up several record types for the domains, each retrieved record type going to its own output field, while the Function's Reverse DNS lookup section retrieves domain names for the two IPs.

The page closes with multi-value results: DNS records can mix single values with multi-value results, and in its example the google.com lookup returns a single record while the cribl.cloud lookup returns four addresses. When a receiver cannot consume the data in array format, one fix is to pick the first result and convert it into a string with this eval, copied from the DNS Lookup Function docs:

Array.isArray(host) ? host[0] : host;

Another option is to randomly pick an address, using a modified version of the eval, also copied from the DNS Lookup Function docs:

Array.isArray(host) ? host[Math.floor(Math.random() * host.length)] : host

Combining them

Disclosure first: the two pages do not document a combined GeoIP-plus-DNS workflow, so this section states only what each page documents independently. What both share is the pipeline convention: each example runs its function as one step in a chain of Stream functions, the GeoIP page chaining a Regex Extract, the GeoIP Function, and an Eval, and the DNS page's example Pipeline chaining an Eval Function and the DNS Lookup Function. Both pages document the same Filter, Description, and Final settings with the same defaults, true, empty, and toggled off, and each function reads a named input field and writes its result into a named output field. Neither page documents a configuration where one function's output feeds the other's input, so any such chain is an operator's own composition of two independently documented functions.

What the docs flag

• In a distributed deployment, the GeoIP database file must be in the same location on the Leader Node and all Worker Nodes, with Lookups Library uploads as the exception, since that selection handles path management automatically.
• In a Cribl.Cloud deployment, uploading the database file to the Lookups Library is currently the only option.
• A reverse DNS lookup is skipped when the field value is not in IPv4 or IPv6 format.
• Making a DNS.lookup() call might degrade performance in unrelated areas of Cribl Stream, so the page says to use this method only if you have not been able to resolve DNS short names using other settings.
• If you enable all of these methods at once and the short name still cannot be resolved after Cribl Stream tries all of them, it will drop the output field from the event.
• Higher DNS cache size limit values increase memory usage (highest allowed: 100000), and a cache time to live of 0 disables cache expiration/refresh behavior.

Frequently asked questions

What database files does the GeoIP Function in Cribl Stream use?

The GeoIP Function works with the binary database from MaxMind's GeoIP or from IPinfo's IP address download. You host the .mmdb database file yourself, on your nodes or uploaded to the Stream Lookups Library.

What happens in Cribl Stream if you leave the DNS Lookup output field name blank?

The lookup result overwrites the original field named in the Lookup field name setting. The page documents this blank-overwrites behavior for both the DNS Lookup Fields section and the Reverse DNS Lookup section.

When does Cribl Stream skip a reverse DNS lookup?

The lookup is skipped when the field value is not in IPv4 or IPv6 format. The docs flag this in a warning note in the Reverse DNS Lookup section.

What are the DNS cache defaults in Cribl Stream?

The cache time to live defaults to 30 minutes, and the cache size limit defaults to 5000. The cache covers both resolved and unresolved lookups, and the page says to use 0 to disable cache expiration/refresh behavior.