Enriching Events with Lookup in Cribl Stream

The Lookup Function in Cribl Stream enriches events with external fields, using lookup table files in CSV, compressed .csv.gz, or binary .mmdb format. The Lookup Function reference page documents every setting the Function exposes, two worked examples, and the deployment caveats worth reading before you wire a lookup into a pipeline.

One scoping note: that page documents exactly one source kind, local files, and its See Also list names the Redis Function for faster lookups using a Redis integration. That is a separate Function and falls outside this post.

What lookup does

Per the docs, the Function takes a Filter expression (JavaScript) that selects the data to feed through it, defaulting to true, meaning it evaluates all events. A Description field carries a free-text note and defaults to empty, and the Final toggle, off by default, stops feeding data to the downstream Functions when it is on.

The core of the configuration is where the lookup data comes from. You select a file you have already uploaded through Cribl Stream's UI at Knowledge > Lookups, or you specify the path directly. Environment variables can be referenced with a dollar sign, and the page shows $CRIBL_HOME/file.csv as its example. One distributed deployment twist: when the field is configured from a distributed deployment's Leader Node, Cribl Stream swaps $CRIBL_HOME/groups/<groupname>/ for $CRIBL_HOME when validating that the file exists, and the default upload path shifts from $CRIBL_HOME/data/lookups on single-instance deployments to $CRIBL_HOME/groups/<groupname>/data/lookups/ on distributed ones.

The backends

The page documents one source family for the Lookup Function: local lookup table files, operated in two modes. The default mode is in-memory. The Use Disk-Based Lookup toggle (default off) switches the Function to a disk-based lookup, and turning it on removes the Match mode and Reload period (sec) options, as they do not apply to disk-based lookups. No remote service backend appears anywhere on the page: everything the Function reads comes from a local file it either holds in memory or addresses on disk. The page's up-front format list is CSV, compressed .csv.gz, and binary .mmdb, while the file path setting itself is labeled Lookup file path (.csv, .csv.gz).

The in-memory mode carries the full settings set:

• Match mode: defines the format of the lookup file and the matching logic, defaults to Exact, and is only available for in-memory lookups; it is disabled when Use Disk-Based Lookup is selected. With Match mode: Regex, patterns run on the JavaScript (ECMAScript) regex engine, and the docs say to pick the ECMAScript (JavaScript) flavor if you test patterns in external tools such as Regex101.
• Reload period (sec): for single-instance deployments, changes to lookup files require an application restart unless a reload interval is enabled. A positive number makes Cribl Stream check for updates and reload the file that often. The -1 default disables reloading.
• Match type: for CIDR and Regex Match modes only, this refines how multiple matches resolve. First match (the default) returns the first matching entry, Most specific scans all entries, finding the most specific match, and All returns all matches in the output, as arrays. The option is not displayed for Exact match mode.
• Lookup fields (.csv): the fields used to key into the lookup table. Lookup Field Name in Event is the exact field name as it appears in events, with nested addressing supported. Corresponding Field Name in Lookup is the field name as it appears in the lookup file, defaults to the in-event value, and is optional.
• Output field(s): the fields added to events after matching the lookup table, defaulting to all if not specified. Each pairs an Output Field Name from Lookup (the field name as it appears in the file) with a Lookup Field Name in Event (defaults to the lookup field name, optional, nested addressing supported), plus an optional Default Value string assigned to the field when the lookup entry is not found.
• Advanced Settings: Ignore case (default off) ignores case when performing Match mode: Exact lookups, and Add to raw event (default off) appends the looked-up values to the _raw field as key=value pairs.

For disk-based lookups the page is shorter but firm on one requirement: the field you reference in the Lookup Function must be indexed, or the lookup will fail, and it points to the Configure Lookups topic, Add Indexes to a Lookup File, for setting that up. It also records that Cribl Edge does not support disk-based lookups, only in-memory lookups, a constraint to weigh before designing a lookup pipeline that might also run on Edge.

The page's first example assigns a sourcetype field to events whose _raw matches a pattern, with Match mode Regex, Match type First match, the event field _raw keyed to the regex column of a table like this one, copied from the Lookup Function examples:

regex,sourcetype
"^[^,]+,[^,]+,[^,]+,THREAT",pan:threat
"^[^,]+,[^,]+,[^,]+,TRAFFIC",pan:traffic
"^[^,]+,[^,]+,[^,]+,SYSTEM",pan:system

In the example's output, the TRAFFIC event gains "sourcetype": "pan:traffic" and the THREAT event gains "sourcetype": "pan:threat".

The second example assigns a location field when destination_ip falls in a CIDR range, with the event field keyed to the range column of this table, also copied from the Lookup Function examples:

range,location
10.0.0.0/24,San Francisco
10.0.0.0/16,California
10.0.0.0/8,US

For the sample event carrying destination_ip 10.0.0.102, both First Match and Most Specific output location as San Francisco, while the All match type outputs an array holding all three matching values in order: San Francisco, California, and US.

Performance notes the page makes

The performance statements the page makes are confined to CIDR match mode. With Match type Most specific, the lookup implicitly searches for matches from most specific to least specific, so there is no need to pre-sort data. With Match type First Match, the docs state it is likely the most performant with large lookups, and offer it as an alternative to Most specific, if the file is sorted with the most specific, relevant entries first; even in that mode it still performs a table scan, top to bottom. Beyond those CIDR notes, the page makes no caching, TTL, or file-size guidance.

What the docs flag

• CSV lookup files are text-based and untyped, so all data in the file is treated as strings, regardless of its intended type, like numbers or dates.
• With Match mode Regex and Match type First match, any empty row in the lookup file, including one at the bottom, causes the Function to always return true.
• Lookups are case-sensitive by default. If the lookup file contains duplicate key names with different values, all Match modes use only the value in the key's final instance, ignoring all preceding instances.
• In distributed deployments, enabling a Reload period can generate conflicts with configuration updates, causing Pipelines to skip executing some Lookup Functions. Cribl recommends enabling it only for lookup files not managed by Cribl Stream's UI, or for files an external process updates, such as a threat list updated via a cron job. For files that are managed by the UI, the distributed deployment overrides the setting as necessary, skipping checks to prevent conflicts. None of those restrictions apply to single-instance deployments.

Frequently asked questions

What file formats can the Cribl Stream Lookup Function read?

The Lookup Function reads lookup table files in CSV, compressed .csv.gz, and binary .mmdb format. Files can be selected from the Knowledge > Lookups area in the Stream UI or addressed by path, and the path can reference environment variables such as $CRIBL_HOME.

How does Cribl Stream reload a local lookup file?

In single-instance deployments, changes to lookup files require an application restart unless a reload interval is enabled. Setting the Reload period (sec) to a positive number makes Stream check for updates and reload the file on that schedule, and the -1 default disables automatic reloading. In distributed deployments the deployment can override the setting to skip checks and avoid configuration conflicts.

What happens when a Cribl Stream lookup file contains duplicate keys?

All Match modes of the Function use only the value in the key's final instance, ignoring all preceding instances in the file. Lookups are case-sensitive by default, unless the Ignore case advanced setting is toggled on for Exact lookups.

Why does a Cribl Stream Regex lookup always return true?

The docs warn that with Match mode Regex and Match type First match, any empty line in the lookup file, including one at the bottom, causes the Function to always return true. Keeping the file free of empty rows is the documented fix.