Routing Data Collected by Cribl Edge

Routing collected data

The Route Collected Data page covers how Cribl Edge processes and routes the data it collects, and it opens with where that data can go. You can route data collected in Cribl Edge in several ways:

• To Cribl Stream: for further processing and scalable distribution. The page points to the Send Data from Cribl Edge to Cribl Stream use case for the detailed flow.
• To Cribl Search: for analysis or visualization, using the Cribl Search Destination.
• Directly to Destinations: such as cheap storage, SIEMs, or other security and IT tools.

The page is blunt about the role boundary: Cribl Edge is not intended for extensive data manipulation. It recommends limiting Pipelines on Edge Nodes to basic Cribl Functions and avoiding complex operations, to prevent bottlenecks and ensure optimal performance. A typical workflow offloads major processing from the Edge Nodes, for example by collecting data locally, filtering out unwanted data with the Drop Function, and routing the remaining data to Cribl Stream for further processing and delivery to your desired Destinations. The page also notes you can check the Pipeline Profiler to access statistics on each Pipeline's performance.

Choosing how the data flows comes down to two interfaces for managing data flow in Cribl Edge: QuickConnect and Data Routes. QuickConnect is described as ideal for simple, straightforward scenarios with a given Source sending the same set of data through an optional Pipeline to one or more Destinations, and it offers ease of use and speedy configuration. Data Routes are the default option, and the page says they are better suited for complex data routing involving conditional processing, providing flexibility, advanced logic, and scalability for handling intricate data flows. The docs suggest QuickConnect for rapid, minimal configuration setups, and Routes when your data requires more sophisticated routing and processing.

Once data is collected, Cribl Edge Pipelines can transform it in pre-processing, processing, or post-processing stages, with common operations like dropping events and enriching them with metadata.

The Drop Function eliminates data from the ingestion pipeline without incurring additional costs. The page lists three ways to take advantage of it: use pre-processing Pipelines to filter and drop specific events, based on criteria like event codes; use processing Pipelines to drop data before cloning events to send them to multiple Destinations, which reduces data volume; and safely experiment with configurations, without incurring costs, by routing data to the DevNull Destination.

For enrichment, the page recommends the Eval Function's Add Field option, which lets you dynamically create new fields based on existing data elements, such as tags from mapping rules, or event metadata. The page's instance: a File Monitor collecting IIS logs from a Windows server, where you could specify a value of index = 'IIS' to create a new field named index with the value IIS, providing essential context for the data in your final analysis system.

Custom tags

The Forward Custom Data with Cribl Edge Tags use case starts from a familiar need: organizations often need to add unique identifiers or metadata to each event forwarded from Cribl Edge to supported Destinations, whether you are managing a small deployment or thousands of nodes, because Edge tags provide the context needed to make sense of your data at scale. The page lists what those identifiers can help with:

• Track the source of events across a large enterprise
• Associate events with specific departments, teams, or business units
• Maintain custom organizational taxonomy in log data
• Enable better filtering and searching in the Destination systems

The guide walks the whole path through a scenario of an operator, Data Detective Dave, enriching data streaming from all Development servers in the Engineering Organization before it is dispatched to a central analytics platform. It covers setting tags during installation, confirming metadata collection, building a pre-processing Pipeline that surfaces the tags as fields, applying it to the right Sources, verifying the output, and keeping the new fields through to the Destination.

The tags themselves are set while installing Cribl Edge. Regardless of the deployment environment (Docker, Linux, Windows, macOS, or Kubernetes), the Bootstrap script via the UI gives you an option to add tags, which help identify and categorize Edge Nodes such as environment type, geographic location, or server role. As you modify the fields within the script interface, the script updates in real-time, reflecting every configuration choice on the screen, including the tags you add. The example sets two tags, Server Role development and Department engineering, so each server wears its identity from the get-go. You can also add tags after you install the Edge Nodes by teleporting directly into the Node.

Tags reach the events through metadata collection. Cribl Edge lets you configure Event metadata sources to enrich events with additional context, and in Edge mode all the Event metadata sources are enabled by default. The Event metadata sources add a __metadata property to every event emitted from enabled sources. The guide still tells you to check that the cribl metadata option is enabled: navigate to Fleet Settings, Limits, Metadata, then use the Event metadata sources drop-down or the Add source button to enable and select metadata sources.

With the metadata flowing, Dave creates a pre-processing Pipeline named host_tags under Fleet, More, Pipelines in the Cribl Edge interface. Pipeline IDs can contain only letters (A-Z, a-z), numbers (0-9), and the characters _ and -. Add an Eval Function to the Pipeline and, in its Evaluate fields section, enter the first row with Name role and Value Expression __metadata.cribl.tags[0], then select Add Field and enter Name department with Value Expression __metadata.cribl.tags[1]. Save the Pipeline.

The Pipeline only matters once it is attached to data. To make sure it runs for all your data, go to Sources, choose your Source, find the Pre-Processing Pipeline option in the Source configuration modal, select host_tags, and save. In the worked scenario, Dave applies it to the Sources feeding data from the Development servers.

Verification uses the Live Data capture on the configured Source. On the Source configuration modal, select Live Data, then Capture, and in the Capture modal, under Where to capture?, select 2. Before the Routes. If the Eval Function is working, the role and department fields appear in the events, and you can confirm them before the data is routed.

The last step is the receiving end. Navigate to Data, Destinations, and configure the Destination so it is not dropping these custom fields; if needed, create a Route that specifically handles the tagged events, and verify that the captured data includes the additional metadata fields you configured.

What the docs flag

Both pages carry caveats worth keeping in mind.

• Edge is not a processing engine. The Route Collected Data page says Cribl Edge is not intended for extensive data manipulation, and it recommends limiting Pipelines on Edge Nodes to basic Cribl Functions and avoiding complex operations to prevent bottlenecks and ensure optimal performance.
• Dropped-event counts are estimates. The page notes that the number of dropped events that lower the ingestion cost is calculated based on average event size, which means it may not represent the exact number of bytes you drop.
• Confirm the metadata source even though it is on by default. The tags page states that in Edge mode all the Event metadata sources are enabled by default, yet the guide still sends you to Fleet Settings, Limits, Metadata to confirm the cribl option, because that source is what carries the tags into the __metadata.cribl.tags array the Eval Function reads.
• Which tag becomes which field is index-driven. In the example, the Eval rows read __metadata.cribl.tags[0] into role and __metadata.cribl.tags[1] into department, so the mapping follows the array index you enter as the Value Expression.
• The Destination can still drop the new fields. The guide ends by checking that the Destination configuration is not dropping the custom fields, and suggests creating a Route that specifically handles the tagged events if needed.

Frequently asked questions

How is data collected by Cribl Edge routed onward?

Cribl Edge can route the data it collects to Cribl Stream for further processing and scalable distribution, to Cribl Search for analysis or visualization, or directly to Destinations such as cheap storage and SIEMs. Data Routes are the default option in Cribl Edge, while QuickConnect covers the simple case of one Source sending the same set of data to one or more Destinations.

How do Cribl Edge installation tags become fields on forwarded events?

The tags you set at install time are gathered by the cribl event metadata source, and they end up in the __metadata.cribl.tags array on events. The use case's Eval Function then copies values out of that array, mapping __metadata.cribl.tags[0] to a role field and __metadata.cribl.tags[1] to a department field.

Which Cribl Edge Pipeline surfaces the custom tags on events?

The use case builds a pre-processing Pipeline named host_tags that contains an Eval Function, and attaches it through the Pre-Processing Pipeline option in the Source configuration. You confirm that the role and department fields appear with a Live Data capture taken Before the Routes.

Should Cribl Edge Pipelines do heavy processing on the collected data?

No, the docs say Cribl Edge is not intended for extensive data manipulation, and they recommend limiting Pipelines on Edge Nodes to basic Cribl Functions. A typical workflow instead collects data locally, drops unwanted events, and routes the remainder to Cribl Stream for further processing and delivery to Destinations.