Version Control and As-Code Configuration for Cribl Stream

Configuration as files

The Set Up Version Control page makes this an installation step: as part of the Cribl Stream installation process for on-prem deployments, verify that Git is installed on the hosts running your deployment. Git is a required component, because it is a dependency for many Cribl Stream features, the Monitoring > Licensing dashboard will display configuration change markers only if you have git installed, and all configuration changes must be committed before they are deployed.

On the deployment side, the Leader notifies Worker Nodes that a new configuration is available, and Worker Nodes pull the new configuration from the Leader Node. The stated requirement is Git 1.8.3.1 or later, and for distributed deployments git must be installed and available locally on the host running the Leader Node.

To verify that git is available and running the required version, run the command the page gives, copied from the version-control page:

git --version

For the deploy step, the page points to the Commit and Deploy Changes documentation for the UI path, and notes that the Cribl API can automate commit and deploy commands too.

Three on-disk configuration files complete the picture: breakers.yml stores Event Breaker configuration, parsers.yml stores configuration data for the Knowledge > Parsers Library, and vars.yml stores configuration data for the Knowledge > Global Variables Library, per the breakers.yml page, the parsers.yml page, and the vars.yml page.

breakers.yml

Event Breaker configuration is stored in breakers.yml, per the breakers.yml documentation. The file holds an Event Breaker ruleset, and the opening of the documented schema, copied from the breakers.yml page, shows the top-level keys:

# ID - ID can contain only letters (A-Z, a-z), numbers (0-9), dashes, underscores, and spaces
# [string; required]
id:
# Library - Library classification for the ruleset
# One of: custom | cribl-custom
# [string; default: custom]
lib:
# Description - Description of the event breaker ruleset
# [string]
description:
# Tags - Tags associated with the ruleset
# [string]
tags:
# Min raw length - The minimum number of characters in _raw to determine which rule to use
# [number; min: 50; max: 100000; default: 256]
minRawLength:

At the top level, id is a required string that can contain only letters (A-Z, a-z), numbers (0-9), dashes, underscores, and spaces. lib is a string, one of custom or cribl-custom, with a default of custom. description and tags are strings. minRawLength is the minimum number of characters in _raw to determine which rule to use; it is a number with a minimum of 50, a maximum of 100000, and a default of 256.

The rule-level block below, copied from the breakers.yml page and continuing where the block above ends, shows the required rule keys:

# Rules - A list of rules that will be applied, in order, to the input data stream
rules:
  # Name - Name of the rule
  # [string; required]
  name:
  # Filter condition - JavaScript expression applied to the beginning of a file or object, to determine
  # whether the rule applies to all contained events.
  # [string; required]
  condition:
  # Event Breaker type - Type of event breaker to use
  # One of: regex | json | json_array | header | timestamp | csv | aws_cloudtrail | aws_vpcflow | azure_flowlog
  # [string; required]
  type:

type is required and is one of regex, json, json_array, header, timestamp, csv, aws_cloudtrail, aws_vpcflow, azure_flowlog. The page scopes several further rule keys to a specific type: eventBreakerRegex, the regex used to break the stream into events at the beginning of the match, is regex type only; jsonArrayField, jsonExtractAll, jsonTimeField, and parentFieldsToCopy are json_array type only; delimiterRegex, fieldsLineRegex, headerLineRegex, cleanFields, and nullFieldVal are header type only; and delimiter, quoteChar, escapeChar, and timeField are csv type only.

Every rule also carries the timestamp settings: timestampAnchorRegex, the regex to match before attempting timestamp extraction, where using the $ end-of-string anchor prevents extraction; the required timestampTimezone; the bounds timestampEarliest and timestampLatest; and the required nested timestamp object, whose type is one of auto, format, current, with length and format alongside. Remaining rule keys on the page: fields, the required maxEventBytes (the maximum number of bytes that an event can be before being flushed), the booleans disabled, parserEnabled, and shouldUseDataRaw, and parser.

parsers.yml

parsers.yml stores configuration data for the Knowledge > Parsers Library, per the parsers.yml documentation. Each parser entry starts with identity keys. The opening of the documented schema, copied from the parsers.yml page:

# Parser ID - Unique identifier for this parser
# [string; required]
id:
# Library - Library classification for the parser
# One of: cribl | cribl-custom | custom
# [string]
lib:
# Description - Description of the parser
# [string]
description:
# Tags - Tags associated with the parser
# [string]
tags:
# Type - Parser or formatter type to use
# One of: csv | elff | clf | kvp | json | delim | regex | grok
# [string; required]
type:

type is required and picks the parser or formatter to use: one of csv, elff, clf, kvp, json, delim, regex, grok. How the parser operates is set by mode: Extract creates new fields, and Reserialize extracts and filters fields and then reserializes, with extract as the documented default. srcField names the field containing text to be parsed, with a default of _raw, and dstField names the destination field.

The mode and srcField entries, copied from the parsers.yml page:

# Fields - Field names to extract
fields:
# Mode - Operation mode - Extract creates new fields. Reserialize extracts and filters fields, and then
# reserializes.
# One of: extract | reserialize
# [string; default: extract]
mode:
# Source field - Field containing text to be parsed

The remaining documented keys describe parsing and output. fields lists field names to extract, cleanFields cleans field names by replacing non [a-zA-Z0-9] characters with _, and fieldFilterExpr, keep, and remove filter the result. Regex parsing has regex, regexList, and fieldNameExpression, plus iterations, the maximum number of iterations for regex matching. Grok parsing has pattern and patternList. Delimited formats have delimiterChar, quoteChar, escapeChar, and nullValue (null fields are not added to events); key and value character sets come from allowedKeyChars and allowedValueChars; delimiter is the field delimiter for CSV parsing; timeField is an optional timestamp field name in extracted events; and overwrite is whether to overwrite existing fields.

vars

vars.yml stores configuration data for the Knowledge > Global Variables Library, per the vars.yml documentation. The documented schema, copied in full from the vars.yml page, is short compared with the other two files:

# Name - Global variable name
# [string; required]
id:
# Library - Library classification for the variable
# One of: cribl | cribl-custom | custom
# [string]
lib:
# Description - Description of the global variable
# [string]
description:
# Type - Data type of the variable
# One of: string | number | encryptedString | boolean | array | object | expression | any
# [string; default: any; required]
type:
# Value - Value or expression for the variable
# [string; required]
value:
# Tags - Tags associated with the variable
# [string]
tags:
# Arguments - For expression type variables, define function arguments
args:

Per entry, id (the variable name) and value are required strings, and type is required, with a default of any. The allowed types are string, number, encryptedString, boolean, array, object, expression, any. value holds the value or expression for the variable, lib classifies it as one of cribl, cribl-custom, custom, and description and tags are strings. args defines function arguments for expression type variables.

What the docs flag

• The Set Up Version Control page conditions its behavior on Git: the Monitoring > Licensing dashboard will display configuration change markers only if you have git installed, and for distributed deployments git must be installed and available locally on the host running the Leader Node.
• The vars.yml page carries one versioned condition: as of Cribl Stream 4.16.0, new Worker Groups do not have sample variables by default.

Frequently asked questions

Does Cribl Stream require Git on the hosts running an on-prem deployment?

Yes, as part of the installation process for on-prem deployments the docs direct you to verify that Git is installed on the hosts running the deployment. Cribl Stream requires Git 1.8.3.1 or later, and for distributed deployments git must be installed and available locally on the host running the Leader Node.

What does breakers.yml store in Cribl Stream?

breakers.yml stores Event Breaker configuration for an Event Breaker ruleset. Each rule's type is one of regex, json, json_array, header, timestamp, csv, aws_cloudtrail, aws_vpcflow, azure_flowlog.

What does minRawLength do in Cribl Stream's breakers.yml?

It is the minimum number of characters in _raw to determine which rule to use. It is a number with a minimum of 50, a maximum of 100000, and a default of 256.

What types can a global variable in Cribl Stream have?

The type key in vars.yml is one of string, number, encryptedString, boolean, array, object, expression, any, with a default of any. For expression type variables, the args key defines function arguments.