Testing Cribl Stream Pipelines with Datagen and DevNull

Why synthetic data

Debugging a pipeline is hard when the traffic you want to observe depends on real senders being available. The Using Datagens page in the Cribl Stream docs puts the purpose plainly: the datagens feature lets you generate sample data "for the purposes of troubleshooting Routes, Pipelines, Functions, and general connectivity."

A synthetic source removes that dependency. You control when events start, you know their shape, and you can stop generation without coordinating with anyone. That is the setup the rest of this post walks through: a Datagen source pushing events through a pipeline, and a DevNull destination that simply drops them, so the whole loop can run without real data reaching anything downstream.

The Datagen source

The Datagen Source page documents a source of type System, with TLS Support listed as N/A and no Event Breaker Support. When a datagen is enabled, each Worker Process uses the specified data generator file to generate events. Those events proceed through Routes and Pipelines, or through a QuickConnect configuration, to configured Destinations, and whichever Worker Process generated an event from the file will also send the same event.

To enable one, select Products on the top bar, then Cribl Stream, and choose a Worker Group under Worker Groups. Two paths are given. Via QuickConnect, navigate to Routing > QuickConnect, select Add Source, and choose either Select Existing or Add New. Via Routes, go to Data > Sources, select the source you want, and select Add Source.

In the New Source modal, General Settings ask for an Input ID, a unique name that identifies this Source definition, with Cribl Stream adding -CLONE to the original Input ID if you clone the Source. There is an optional Description, and there is the Datagens list itself.

Each entry in the Datagens list names a data generator file and sets Events per second per Worker Node, the maximum number of events to generate per second, per Worker Node or Edge Node, which the page says defaults to 10.

Under Optional Settings you can add Tags, separated by a tab or hard return between arbitrary tag names. The page notes these tags are for filtering and grouping in the Cribl Stream UI and are not added to processed events.

Processing settings come next. In the Fields section you can define new fields or modify existing ones using JavaScript expressions, similar to the Eval function. The Field Name is either a new field, unique within the event, or an existing field whose value you modify, and the Value is a JavaScript expression, enclosed in quotes or backticks, that computes the field's value or can be a constant. In Pre-Processing, a Pipeline drop-down lets you select a single existing Pipeline or Pack to process data from this input before the data is sent through the Routes.

Advanced Settings expose an Environment field for GitOps: specify a single Git branch on which to enable the configuration, and if the field is empty, the config is enabled everywhere.

Under Connected Destinations you choose how the source's data goes on. Selecting Send to Routes enables conditional routing, filtering, and cloning of this Source's data via the Routing table. Selecting QuickConnect sends it to one or more Destinations via independent, direct connections. The page also documents one internal field for this source, __inputId: meta fields are not part of an event, but they are accessible, and Functions can use them to make processing decisions.

The Using Datagens page covers the files the source consumes. Several datagen template files ship with the product, out of the box, and you can create others from sample files or live captures. Its tutorial enables two of those out-of-the-box generators: apache_common.log set at 4 Events Per Second Per Worker Node, and syslog.log set at 8. To build your own template, it has you paste a sample through Import Data in the right preview pane of the Data Routes or Pipelines page, select an Event Breaker from the drop-down so the pasted text breaks into individual events and timestamps extract correctly, then select Create a Datagen File. You enter a file name, such as vpc-flow-datagen.log, and confirm the timestamp template format, ${timestamp: %s}. The page explains that ${timestamp: <format>} is a template the datagen engine uses to insert the current time, in each newly generated event, using the given format, with %s the desired strftime format for the epoch. You save the file and confirm it in the Datagens tab of the preview pane. The same tab is where you modify a datagen: the edit button offers edit, clone, and delete, plus changes to metadata such as File name, Description, Expiration time, and Tags, and Edit Datagen exposes the raw data the datagen uses to generate events.

The tutorial's sample is a set of AWS VPC Flow log lines, copied from the Using Datagens page:

2 123456789010 eni-abc123de 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK
2 123456789010 eni-abc123de 172.31.9.69 172.31.9.12 49761 3389 6 20 4249 1418530010 1418530070 REJECT OK
2 123456789010 eni-1a2b3c4d - - - - - - - 1431280876 1431280934 - NODATA
2 123456789010 eni-4b118871 - - - - - - - 1431280876 1431280934 - SKIPDATA
2 123456789010 eni-1235b8ca 203.0.113.12 172.31.16.139 0 0 1 4 336 1432917027 1432917142 ACCEPT OK
2 123456789010 eni-1235b8ca 172.31.16.139 203.0.113.12 0 0 1 4 336 1432917094 1432917142 REJECT OK
2 123456789010 eni-f41c42bf 2001:db8:1234:a100:8d6e:3477:df66:f105 2001:db8:1234:a102:3304:8879:34cf:4071 34892 22 6 54 8855 1477913708 1477913820 ACCEPT OK

The DevNull destination

The DevNull Destination page is short, and its opening is the whole pitch: the DevNull Destination simply drops events, and Cribl provides it as a basic output to test Pipelines and Routes. The page's metadata block lists the type as Internal, with TLS Support N/A and PQ Support N/A.

There is no configuration to speak of: the page states that a DevNull Destination is preconfigured and active as soon as you install Cribl Stream. It then walks through verifying that. Select Products on the top bar, then Cribl Stream, and choose a Worker Group under Worker Groups. Via QuickConnect, navigate to Routing > QuickConnect on Stream, or Collect on Edge, where a DevNull Destination is preconfigured and ready to use in the right Destinations column. Via Routes, select Data > Destinations or More > Destinations on Edge, select the Destination you want, and click anywhere on the devnull row to proceed.

A test loop

Put the two pages together and you have a loop with both ends under your control, in the order the docs support:

1. Build the source side. Add a Datagen Source for a Worker Group, give it an Input ID, pick a data generator file, and set its Events per second per Worker Node, which defaults to 10. Select Save, then Commit and Deploy.
2. Wire the drain. In QuickConnect, the DevNull Destination is preconfigured and ready to use in the right Destinations column. On the Routes side, the devnull row sits on the Destinations page, and clicking anywhere on it proceeds.
3. Start generation. When the datagen is enabled, each Worker Process uses the generator file to produce events, those events proceed through Routes and Pipelines, or through a QuickConnect configuration, to configured Destinations, and the Worker Process that generated an event sends it. At the other end, the DevNull destination simply drops each event.
4. Confirm the flow. In the sidebar, select Monitoring, select Data > Sources, and search for datagen.

That is the whole loop: synthetic events in, everything dropped on the way out. Nothing downstream sees the traffic, which is what makes it a safe place to check a Route, a Pipeline, or a Function before real data flows through it.

Frequently asked questions

What does the Cribl Stream DevNull destination do with events?

The DevNull Destination simply drops events, which is why Cribl provides it as a basic output to test Pipelines and Routes. It requires no configuration, and a DevNull Destination is preconfigured and active as soon as you install Cribl Stream.

What event rate does a Cribl Stream datagen use by default?

The Events per second per Worker Node field defaults to 10, which is the maximum number of events to generate per second, per Worker Node or Edge Node. The tutorial in this post sets one file at 4 and another at 8.

Do I need to write datagen files myself in Cribl Stream?

No. Several datagen template files ship with the product, out of the box, and you can create others from sample files or live captures. You can also build a template from pasted sample lines after selecting an Event Breaker and confirming the timestamp template format, then saving the datagen file.

How do I confirm a Cribl Stream datagen source is generating data?

In the sidebar, select Monitoring, select Data > Sources, and search for datagen. That confirms the source side of the loop, whose events proceed through Routes and Pipelines, or through a QuickConnect configuration, to configured Destinations.