Automating Cribl Stream with the CLI

What the CLI covers

Cribl Stream's CLI is ./cribl, and the docs cover it one command at a time. This post works through four of those commands, one page each: status, pipe, mode-worker, and git. Between them they cover checking that an instance is healthy, testing data through a Pipeline, enrolling an instance as a Worker Node, and committing Worker Group configuration changes. Each page's header points to the full documentation index (llms.txt); this post limits itself to what those four pages document.

In practical terms, status is your probe, pipe is your test harness, mode-worker is how a node joins the cluster, and git is how configuration moves under version control. Because each is a single command, all four drop straight into shell scripts and scheduled jobs. Every command sample, option table, and line of sample output below comes from those four pages.

Checking status

The status command "displays status of Cribl Stream, including the API Server address, instance's mode (Leader or Worker), process ID, and GUID. Executes immediately." The usage is:

./cribl status

The sample response from the docs:

Address: http://172.17.0.3:9000
Mode: master
Status: Up
Software Version: 3.1.0-f765e418
Config Version: 347079c
Master: 0.0.0.0:4200
PID: 4100
GUID: e706052a-ace9-4511-a7c7-b58a414a07d3

Every line is a key and a value: the API Server address, the instance mode, the status, the software and config versions, the master endpoint, the process ID, and the GUID. The values on your instance will differ, but the shape of the block is what a script parsing status from the shell gets back. Note the sample prints Mode: master; the page's description says the mode reported is Leader or Worker, so read the mode line against your own deployment.

Piping events through a pipeline

The pipe command "feeds standard input (stdin) to a Pipeline." By default, cribl pipe expects newline-delimited JSON (NDJSON): each line is one serialized event object that becomes an event, typically including _raw. The documented usage:

cat sample.log |  ./cribl pipe -p pipelineName
cat sample.log |  ./cribl pipe -p pipelineName 2>/dev/null

Pass -e / --breaker and stdin is treated as raw data instead, and your chosen Event Breaker Ruleset breaks the stream into events before the Pipeline runs. The Ruleset must exist in the active configuration; if it does not, the command fails and the error names the missing ID. Pass -f / --fields to add or overwrite fields on each event using the same name / value shape as Source Fields, where each value is a JavaScript expression evaluated on the event. When you use -f with -e, those fields are applied before breaking so rule conditions in the Ruleset can evaluate against them.

The docs show one full example of raw stdin through an Event Breaker Ruleset, with Source-style fields (shell-escape the value you pass to -f as needed):

cat api_dump.json | ./cribl pipe -p packPipeline -a myPack -e "My API Ruleset" -f '[{"name":"index","value":"\"myindex\""},{"name":"sourcetype","value":"\"mysourcetype\""}]'

When you use -a / --pack for Pack context, Event Breaker Ruleset IDs are resolved in that Pack (and the default context) the same way as in the UI. The sample response from a run, copied from the docs' ... abridged output, shows the pipeline loading, functions initializing, and stdin events starting to push:

{"time":"2021-08-20T20:37:00.017Z","cid":"api","channel":"commands","level":"info","message":"creating new pipeline","id":"main","conf":{"asyncFuncTimeout":1000,"functions":[{"id":"eval","disabled":false,"filter":"true","conf":{"add":[{"name":"cribl","value":"'yes'"}],"remove":[]}}]}}
{"time":"2021-08-20T20:37:00.019Z","cid":"api","channel":"pipe:main","level":"info","message":"start loading and initializing functions","count":1}
{"time":"2021-08-20T20:37:00.021Z","cid":"api","channel":"pipe:main","level":"info","message":"finished loading and initializing functions","count":1}
{"time":"2021-08-20T20:37:00.022Z","cid":"api","channel":"commands","level":"info","message":"START pushing stdin events","id":"main"}
{"time":"2021-08-20T20:37:00.028Z","cid":"api","channel":"GrokMgr","level":"info","message":"loaded grok patterns","count":152}

The full set of documented options:

• -p <pipeline>: Pipeline to feed data through.
• -d: Include dropped events.
• -c <cpuProfile>: Perform CPU profiling.
• -t: Perform Pipeline tracing.
• -a <pack>: Optional Cribl Pack context. Mutually exclusive with -b.
• -b <project>: Optional Cribl Project context.
• -e <ruleset>: Event Breaker Ruleset ID to apply to raw stdin. If you omit this flag, stdin is read as NDJSON.
• -f <jsonArray>: A JSON array of field objects with name and value keys, applied before the Pipeline runs. The argument must parse as valid data whose top-level value is an array; if it does not, the command fails before events are processed.
• -i: Runs the pipe command without sandboxing JavaScript expressions from potential attackers.

Worker mode from the command line

The mode-worker command configures an instance as a Cribl Stream Worker Node. Read this behavior before the flags: running this command resets all the related options, so even if you want to change only one option, you also need to provide values for all the other arguments. And when you generate bootstrap scripts through the UI to add or update Workers, those scripts automatically set the -S option to enable TLS on the Leader Node.

The documented usage, with the sample token redacted:

./cribl mode-worker -H 127.0.0.1 -p 9420 -u <your-token>

The -H <host> -p <port> -u <authToken> parameters are required for a successful connection to the Leader, per an info box on the page. The full argument set:

• -d <deploymentId>: Deployment ID for reporting telemetry on multiple deployments.
• -H <host>: Leader's hostname or IP address. Required.
• -p <port>: Leader's cluster communications port. The docs' example is 4200. Required.
• -n <certName>: Name of the saved certificate to use. Mutually exclusive with -k or -c.
• -k <privKeyPath>: Path on the server to the private key to use. Accepts PEM format. Can reference $ENV_VARS.
• -c <certPath>: Path on the server to the certificate to use. Accepts PEM format. Can reference $ENV_VARS.
• -u <authToken>: Authentication token to include as part of the connection header. Required.
• -S <tls>: Sets, or disables, secure communication between Leader and Worker/Edge Nodes via TLS. Accepts true or false.
• -e <envRegex>: Regex to select environment variables to report to Leader. The docs' example is /test/.
• -t <tags>: Tag values to report to Leader.
• -g <group>: Worker Group to report to Leader.

Where the token comes from depends on the deployment. In Cribl.Cloud, get the token for your Workspace from the Worker Node modal (Add/Update Worker Node) in the Auth token field. In an on-prem deployment, get it from Settings > Global > System > Distributed Settings > Leader Settings > Auth token. For the full cluster-side setup, the page points to the Set Up Leader and Worker Nodes page.

Configuration under git

The git command manages Worker Group configuration. It exposes four sub-commands: git commit, git commit-deploy, git deploy, and git list-groups.

git commit commits Worker Group or Fleet configuration changes. The docs show:

./cribl git commit -g production_logs -m "Add syslog pipeline"
./cribl git commit -g webserver_data -m "Remove Parser function from webhook pipeline"

Its arguments are -g <group>, the Group ID, and -m <message>, the commit message.

git commit-deploy commits Worker Group or Fleet configuration changes and also deploys them, taking the same -g and -m options:

./cribl git commit-deploy -g production_logs -m "Add syslog pipeline"
./cribl git commit-deploy -g webserver_data -m "Remove Parser function from webhook pipeline"

git deploy deploys Worker Group or Fleet configuration changes. Its arguments are -g <group>, the Group ID, and -v <version>, the version to deploy. The usage example and sample response, reproduced exactly as printed on the docs page:

./cribl git commit-deploy -g production_logs -v 7c04de1

Successfully committed version 7c04de1

git list-groups lists Worker Groups or Fleets. The docs list it as the fourth sub-command; the page gives it a one-line description and no usage example or argument table.

That is the scope this post claims. Probe with status, test with pipe, enroll with mode-worker, and move configuration with git: four plain commands, each documented page by page, each runnable without a browser.

Frequently asked questions

What does the cribl status command display in Cribl Stream?

It displays the status of Cribl Stream, including the API Server address, the instance mode (Leader or Worker), the process ID, and the GUID. The command executes immediately, and the output is one key and value per line.

What input format does cribl pipe expect by default?

By default it expects newline-delimited JSON, where each line is one serialized event object that becomes an event. With the -e breaker flag, stdin is treated as raw data instead, and an Event Breaker Ruleset breaks the stream into events before the Pipeline runs.

Why does cribl mode-worker require values for all the arguments?

Running the command resets all the related options, so even if you want to change only one option, you also need to provide values for all the other arguments. The -H, -p, and -u parameters are required for a successful connection to the Leader.

Which subcommands does the Cribl Stream git command have?

The docs document four: git commit, git commit-deploy, git deploy, and git list-groups. Commit and commit-deploy take a group ID and a commit message, and deploy takes a group ID and a version to deploy.