Monitoring Cribl Stream With Cribl Stream

Why the router should watch itself

A routing box is part of every monitoring pipeline, so it needs monitoring of its own. Cribl Stream ships two built-in Sources for exactly that. The Cribl Internal Source enables you to "capture and send Cribl Stream's own internal logs and metrics through Routes and Pipelines," while the System Metrics Source lets a Stream instance "collect metrics from their hosts." Point both at your monitoring destination and the router reports its throughput, operational logs, and host health to the same store as everything else.

Both are fixed in shape: you can't configure custom Cribl Internal Sources, and neither lists TLS or Event Breaker support. What varies is granularity, detail level, and destination.

The Cribl Internal source

Inside the Cribl Internal Source you enable one of two options as a Source, CriblLogs or CriblMetrics, from QuickConnect or the Routes page; the Input ID is prefilled with the option's name and cannot be changed.

Per the Cribl Internal Source documentation, the CriblMetrics option "offers Cribl's most up-to-date and precise view of event throughput and transformation, aggregated every 2 seconds at the Worker Process level." CriblLogs is less granular, tracking logs that are written once per minute, so the page warns that it might fail to reflect Worker Process crashes or restarts.

In Distributed mode the CriblLogs option processes internal logs only from Worker Processes; logs on the Leader remain on the Leader, because the Leader Node is not part of any processing path. In both Distributed and Single-instance mode, CriblLogs omits API Process logs (telemetry and license-validation traffic); for API Server or Worker Group events the page points to a Script Collector.

Two knobs steer CriblMetrics volume: the Metric name prefix (default cribl.logstream.) and the Full fidelity toggle, which off reduces CPU load by excluding metrics for fields listed in General Settings > Limits > Metrics > Disable field metrics. With Full fidelity on, Cribl Stream generated metrics include the set copied from the Cribl Internal Source page:

cribl.logstream.host.(in_bytes,in_events,out_bytes,out_events)
cribl.logstream.index.(in_bytes,in_events,out_bytes,out_events)
cribl.logstream.project.(in_bytes, in_events, out_bytes, out_events)
cribl.logstream.source.(in_bytes,in_events,out_bytes,out_events)
cribl.logstream.sourcetype.(in_bytes,in_events,out_bytes,out_events)

Internal metrics are generated every 2 seconds by default; the page's lever for longer intervals is the cribl_metrics_rollup Pipeline that ships with Cribl Stream, attached to the Source as a pre-processing Pipeline. Its Rollup Metrics Function carries a default Time Window of 30 seconds, adjustable to a different granularity.

In the same cribl_metrics_rollup pre-processing Pipeline or a clone, the final Eval Function removes the sourcetype attribute from metrics events, leaving only event_sourcetype and preventing duplicate sourcetype events from being routed to Destinations, using the Filter expression on the Cribl Internal Source page:

_metric && _metric.startsWith('cribl.logstream.sourcetype.')

The Source adds internal fields for routing its output: source is cribl, and host is the hostname of the Cribl instance. CriblMetrics operates on metrics that Worker Nodes report to their Leader Nodes, typically including source and host; because a downstream service such as Splunk might overwrite them, the Source copies the values into event_ prefixed dimensions: event_source and event_host. If no downstream service overwrites source or host, drop the added dimensions with an appropriate Function or suppress them via the Metrics limits.

System metrics

The System Metrics page documents host collection on Linux (and, in Cribl Edge, on macOS), and notes that Stream can "populate some standard metrics dashboards right out of the box."

The default Source is prefilled with Input ID in_system_metrics, unchangeable via the UI; cloning adds -CLONE to the original Input ID. The Polling interval is how often to collect metrics, in seconds, defaulting to 10.

Host Metrics offer four buttons per event type: Basic (minimal, averaged or aggregated), All (full, detailed, per individual CPUs and interfaces), Custom (a level per event type), and Disabled (no metrics). In Basic mode the page documents: System captures load averages, uptime, and CPU count; CPU captures active, user, system, idle, and iowait percentages over all CPUs; Memory captures total, used, available, swap_free, and swap_total; Network captures bytes, packets, errors, and connections over all interfaces; Disk captures disk used in percent, bytes read and written, and read and write operations over all mounted disks.

Custom level adds Per CPU, Per-interface, and Per-device metrics, type filters for Network and Disk, Detailed metrics, and, for Network, a Protocol metrics toggle covering ICMP, ICMPMsg, IP, TCP, UDP, and UDPLite.

Process Metrics is a separate mechanism. Create a Process Set with a JavaScript Filter Expression over process object fields, and matching processes are returned as individual events. Process-specific metrics are not affected by the Host Metrics detail setting, and once at least one Process Set is saved, Cribl Stream collects them at the Polling interval, with the __process_set internal field recording which set owns each metric. The System Metrics page offers this example, looking for Java processes in the sleep state using more than 100% of CPU or more than 25% of memory:

cmdline.args[0] == 'java' && status == 'S' && (cpu > 100 || memory_percent > 25)

Container Metrics reports the server event plus per running container events at Basic, adding per-device and detailed metrics at All; Docker socket defaults to /var/run/docker.sock and /run/docker.sock, and Docker timeout is 5 seconds. GPU Metrics track Nvidia GPU usage; Basic captures temperature, GPU utilization, and memory utilization, and require the Nvidia drivers installed together with the nvidia-smi utility.

Cribl has shared a Prometheus dashboard for Cribl Stream System Metrics in the public Grafana library, suitable for Basic mode; the Node Exporter dashboard handles the detailed case with the prometheus_metrics pre-processing pipeline and All mode.

Building a self-health view

Stay inside what the two pages document and the self-health picture has three layers.

Event level, from CriblMetrics: with Full fidelity on, in_bytes, in_events, out_bytes, and out_events per host, index, project, source, and sourcetype, aggregated every 2 seconds at the Worker Process level and coarsened to a 30-second default window by the cribl_metrics_rollup Pipeline.

Operational state, from CriblLogs: the once-per-minute internal log stream, which the page says might fail to reflect Worker Process crashes or restarts; in Distributed mode it covers Worker Processes, not the Leader.

Host level, from the System Metrics Source in Basic: load averages, uptime, CPU count, CPU percentages, memory including swap_free and swap_total, network bytes, errors, and connections, and disk usage with its read and write volumes. Process Sets add specific-process events such as the Java example, attributed by __process_set.

Route each Source to your monitoring destination through the Routing table, whose Send to Routes mode adds conditional routing, filtering, and cloning. Keep the layers separable with the internal fields: source is cribl, host identifies the instance, and event_source and event_host preserve originals when a destination overwrites them.

What the docs flag

• You can't configure custom Cribl Internal Sources.
• CriblLogs omits API Process logs (telemetry and license-validation traffic) in both Distributed and Single-instance mode; the page points to a Script Collector for API Server or Worker Group events.
• In Distributed mode, CriblLogs processes internal logs only from Worker Processes; the Leader is excluded.
• In Cribl.Cloud on Cribl-managed Worker Groups, the CriblLogs Source contains only logs related to Sources and Destinations; the System Metrics Source is available on Cribl.Cloud on customer-managed hybrid Workers, but not on Cribl-managed Workers.
• CriblLogs tracks logs written once per minute and might fail to reflect Worker Process crashes or restarts.
• A warning on the internal page: all Cribl internal fields are subject to change and modification; Cribl provides them to assist with analytics and diagnostics but does not guarantee that they will remain available.

Frequently asked questions

What does the Cribl Internal Source in Cribl Stream monitor?

It captures Cribl Stream's own internal logs and metrics and sends them through Routes and Pipelines via two options, CriblLogs and CriblMetrics. CriblMetrics aggregates metrics at the Worker Process level every 2 seconds, and the shipped cribl_metrics_rollup Pipeline can roll that interval up to 30 seconds by default. You can't configure a custom Cribl Internal Source.

How often does the Cribl Stream System Metrics Source poll by default?

The Polling interval defaults to 10 seconds. The same interval is what process metrics for a saved Process Set follow.

Which platforms does the Cribl Stream System Metrics Source cover?

In Cribl Stream it collects host metrics on Linux, and on macOS in Cribl Edge. In Cribl.Cloud the Source is available on customer-managed hybrid Workers but not on Cribl-managed Workers. GPU metrics additionally require the Nvidia drivers installed together with the nvidia-smi utility.

How can I reduce the volume of internal metrics from the Cribl Internal Source?

Attach the cribl_metrics_rollup Pipeline that ships with Cribl Stream as a pre-processing Pipeline; its Rollup Metrics Function carries a default Time Window of 30 seconds that you can adjust. Alternatively, turn off Full fidelity to exclude metrics for the fields listed under Disable field metrics, which reduces CPU load. You can also drop the event_source and event_host dimensions when no downstream service overwrites the original fields.