The Parser Function in Cribl Stream

What the function does

The Parser Function page in the Cribl Stream documentation describes a Function that extracts fields from events or reserializes events with a subset of fields, and reserialization preserves the event format. Choose a parser type when you know the data format, or use Auto-Parse to detect the format and extract fields with the same Auto-Datatyping logic used in Cribl Search.

In its comma-delimited example, when fieldA and fieldB are filtered out, those fields' positions are set to null but not deleted completely. The Parser Function cannot remove fields it did not create, while a subsequent Eval Function can.

With Auto-Parse toggled on, the Function detects a matching Datatype for the event, extracts top-level fields from _raw using the parser for that Datatype, and strips Search-internal metadata fields from the event. It adds two fields: datatype, set to the ID of the detected Datatype or to Uncategorized when no Datatype matches, in which case no fields are extracted from _raw, and isParsed, set to true. The page says to keep the two fields together: they prevent a downstream Source, Route, Pipeline, or Pack from automatically parsing the event again.

How it is configured

The Usage section of the Parser Function page documents the settings below.

• Filter: Filter expression (JavaScript) that selects the data to feed through the Function. Defaults to true, meaning it evaluates all events.
• Description: Simple description about this Function. Defaults to empty.
• Final: Toggle on to stop feeding data to the downstream Functions. Default is toggled off.

Operation mode has two modes: Extract creates new fields, and Reserialize extracts, filters fields, and then reserializes. Auto-Parse detects the event format and extracts fields from _raw. The toggle is on by default and sets Operation mode to Extract; toggle it off to select a specific Type.

The Type setting selects the Parser/Formatter type to use: CSV, Extended Log File Format, Common Log Format, Key=Value Pairs, JSON Object, Delimited values, Regular Expression, and Grok. Two additional options appear in the drop-down but can only be used when Operation mode is set to Extract, and both expand the UI: Grok and Regular Expression.

The remaining settings, as documented on the page: Library selects an option from the collection of available Parsers; specifying a Library auto-generates an example list of fields, which the page says may still need to be modified. Source field is the field containing text to be parsed, not usually needed in Reserialize mode. Destination field names the field that receives the extracted and serialized fields; if it already exists, it is overwritten, and if multiple new fields are created and it is populated, the new fields become elements of an array named by its value. Clean fields appears for Type: K=V Pairs; toggled on, it replaces non-alphanumeric characters in field names with _ and strips leading and trailing " symbols.

List of fields names the fields expected to be extracted, in order; if not specified, Parser auto-generates fields. The page says the list will not display when Type is set to JSON Object or Key=Value Pairs; it is only relevant for structured formats like CSV, CLF, and ELFF.

Three settings decide which parsed fields survive. Fields to keep: specifying any fields here makes the Parser keep only those and drop all other parsed fields, even if Fields to remove is empty, and it takes precedence over Fields to remove and Fields filter expression. Wildcards and nested addressing work, so _raw* refers to _raw and all nested fields inside it. Fields to remove: a list of fields to remove; supports wildcards; cannot remove fields matching Fields to keep. Fields filter expression: evaluated against each field's {index, name, value} context; return truthy to keep, falsy to remove; index is zero-based.

With Type set to Delimited Values, four extra options appear, each with a default: Delimiter, the character to split the value on, defaults to comma (,); Quote char, the character used to quote literal values, defaults to "; Escape char, the character used to escape the delimiter, defaults to \; and Null value, the field value representing the null value and omitted, defaults to -. Setting Type to Grok displays the Grok Function UI; the same passage names the regex type Regex Extract and points it at the Regex Extract UI, while the type list names the type Regular Expression.

Two Advanced Settings: Allowed key characters accepts characters permitted in a key name even though they are normally separator or control characters, entries separated by a tab or hard return, and it does not affect how the value is parsed. Allowed value characters does the same for a value name, and does not affect how the key is parsed.

Auto-Parse changes what is configurable: because automatic parsing always reads from _raw, Source field, Destination field, and Library are not available when the toggle is on, and custom parsers from Knowledge > Parsers are not supported.

Example from the page

The page's Examples section shows creating and dropping fields, reserialization, and serializing to CSV versus JSON. Example 1 inserts this sample using Sample Data > Import Data, copied from the Parser Function page:

2019/06/24 05:10:55 PM Z a=000,b=001,c=002,d=003,e=004,f=005,g1=006,g2=007,g3=008

The test Function sets Type to Key=Value Pairs. Scenario A keeps fields a, b, c and drops the rest, with Fields to keep at a, b, c and both other settings empty: the event ends with a: 000, b: 001, c: 002, plus cribl_pipe: parser2. Scenarios B to E vary the Fields filter expression, including name.startsWith('g') and name.startsWith('g') && value=='007', and use index, which the page defines as the zero-based location of a field in the array of all fields extracted by this Parser.

Example 5 uses Type Grok on an event, copied from the Parser Function page:

2020-09-16T04:20:42.45+01:00 DEBUG This is a sample debug log message

The Pattern the page says will work, copied from the Parser Function page:

%{TIMESTAMP_ISO8601:event_time} %{LOGLEVEL:log_level} %{GREEDYDATA:log_message}

Two more examples rest on screenshots: Example 2 reserializes a JSON event, removing the level field only if set to info, and removing startTime and fields in the values.total. path that end in Cxn. Example 3 serializes fields to CSV or JSON using a second Parser Function in Serialize operation mode.

What the docs flag

In the page's Characters to Escape section, when Type is Key=Value Pairs, field values containing an = delimiter must be surrounded by quotes, or the Function returns unexpected results. The page's example of four fields and values, with the two values containing = quoted, copied from the Parser Function page:

one=1,two="value=2",three=3,four="value=4"

With Type set to CSV, the rule runs parallel: column values containing quotes or commas must quote those characters.

Auto-Parse carries the page's strictest version gate: version 4.19 or later on every Worker Node, no Worker Node older than the Leader, and all Worker Nodes in a Worker Group on the same version. A Worker Node that does not meet these requirements skips the Function and passes events through unparsed; no error appears in the UI, Pipeline preview extracts fields as expected because preview runs on the Leader, and only the Worker Node's own logs record the failure.

Automatic parsing has lower throughput than a Parser Function with a specific Type: in Cribl performance tests with no changes to infrastructure, a specific Type delivered 3 to 5 times the throughput of automatic parsing. Each Pipeline that uses Auto-Parse adds more than 10 MB of memory per Worker Process, and multiple Pipelines multiply that overhead.

Removing datatype or isParsed before another automatic parsing operation causes Cribl to evaluate the event again, which reduces throughput; the page's remedy is an Eval Function after the last automatic parsing operation when those fields must not reach a Destination. And fields lists with negated terms are order-sensitive: the page's example says !foobar, foo* means all fields that start with foo, except foobar.

Frequently asked questions

What does the Parser Function in Cribl Stream do?

It extracts fields from events, or reserializes events with a subset of fields, and reserialization preserves the event format. With Auto-Parse toggled on, it detects the format and extracts top-level fields from _raw with the same Auto-Datatyping logic used in Cribl Search.

What fields does the Cribl Stream Parser Function add when Auto-Parse is on?

It adds datatype, set to the ID of the detected Datatype or to Uncategorized when no Datatype matches, and isParsed, set to true. These fields stop a downstream Source, Route, Pipeline, or Pack from automatically parsing the event again.

Which parser types does the Cribl Stream Parser Function page list?

The page's options are CSV, Extended Log File Format, Common Log Format, Key=Value Pairs, JSON Object, Delimited values, Regular Expression, and Grok. Grok and Regular Expression appear in the drop-down but can only be used when Operation mode is set to Extract, and both expand the UI.

How do the three field settings interact in the Cribl Stream Parser Function?

Fields to keep takes precedence over Fields to remove and Fields filter expression, and fields matching Fields to keep cannot be removed by Fields to remove. Listing any fields in Fields to keep makes that list the complete set of parsed fields to retain, so unlisted parsed fields are dropped even when Fields to remove is empty.