Cribl Search Datasets and Rules

Cribl Search organizes what it indexes in three moves: Datasets group the data, Dataset rules decide where events land, and Datatype rules shape logs as they arrive.

Datasets

A Dataset in Cribl Search is a named container that groups related data. The Create Search Datasets page distinguishes two kinds. Search Datasets organize data ingested into Cribl-hosted lakehouse engines and are optimized for fast, schema-aware searches and AI workflows; each stores either logs or metrics, not both. Federated Datasets instead query your own storage, such as S3 or Azure Blob, in place, with no ingestion or indexing.

Cribl-hosted Search Datasets come in two flavors. Log Datasets hold parsed log events; you add them yourself, except the fallback main Dataset, and one can hold logs from multiple Sources and of different Datatypes. Retention runs from 1 day to 10 years, defaulting to 1 year, and Datasets auto-scale with routed volume and retention. metrics Datasets hold metric time series; Cribl Search provides one per lakehouse engine.

main appears with your first lakehouse engine as a catch-all for unrouted events. The _dataset_reason field records why: no_rules_matched when no rule matched, invalid_dataset_matched (<datasetId>) for rules pointing at a deleted or invalid Dataset, and invalid_dataset_provided when an event's dataset field points at an invalid Dataset. You can change its retention, but main can't be deleted.

Each metrics Dataset is auto-provisioned per engine: the first is named metrics, later engines get metrics_<engine_id>. It can't be deleted or renamed. Retention runs from 1 to 365 days, defaulting to 365. Datatypes don't apply, because every metric is stored in the same time series shape.

You add a Log Dataset under Products > Search > Data > Datasets > Add Dataset, with a unique ID, a Description, and Tags. The next step sets retention (1 day to 10 years, default 1 year), the storing lakehouse Engine, and the expected timestamp window: Earliest expected timestamp, a relative time expression, and Latest expected timestamp, up to 7d, defaulting to 1d. Performance > Add Field lists accelerated fields; the docs recommend no more than 15, since they add ingest and index costs.

That window matters at ingest: out-of-range logs are retimestamped, with _time reset to now() and the original value preserved in _original_time, while out-of-range metrics are rejected. Default latest bound: one day after now() for logs, 10 minutes for metrics.

You can also send results into a Log Dataset with the export operator and search keyword, suggested for materializing a federated search or enrichment pipeline once, then re-searching the cached results. Copied from the Create Search Datasets page:

dataset="cribl_search_sample"
| export to search mySearchDataset

The target must be a Log Dataset whose engine is in Ready status; only Admin and Editor Search Members can run export. Clear Dataset wipes data but keeps routing rules and retention; Delete Dataset removes a Log Dataset's data and configuration. Both need Maintainer access, can't be undone, and metrics Datasets can't be deleted.

Dataset rules: organizing data

Dataset rules control which data lands in which Dataset. Per the Organize Data with Dataset Rules page, each rule captures data matching a KQL expression and sends it to a Search Dataset: Log Dataset Rules match log events and route into a Log Dataset you created, while Metric Dataset Rules match metrics and route into your engine's metrics Dataset. Rules run top-down, and the first match wins.

The Log Rules and Metric Rules tabs appear only with the Metric Rules Preview enabled; otherwise rules list without tabs. Both live under Products > Search > Data > Get Data In > 3. Dataset Rules.

To add a Log Dataset rule, select Add Dataset Rule, name and describe it, and enter a KQL expression in Kusto expression to match. Send data to picks the target Log Dataset or Drop; the optional Modify fields step runs a KQL extend expression for Dataset-specific enrichment. Check Enabled, confirm with Add, and drag to reorder, keeping specific rules above broader ones. Events matching no rule, or a rule pointing at a deleted Dataset, fall back to main.

Point expressions at datatype, assigned through Datatyping, and __inputId, the Source identifier in type:id form (supported types include syslog and splunk_hec); any other parsed field works too. KQL must evaluate to true or false; case-insensitive =, wildcards, and pipes into where, find, or search all work, but aggregating or reshaping operators like stats or project, and let or set, aren't allowed. The page's syslog example, copied from the Organize Data page:

datatype = "syslog_rfc3164" and __inputId = "syslog:my_source_id"

The page's Modify fields example converts a bytes field to megabytes, copied from the Organize Data page:

size_mb = bytes / 1048576.0

An override path exists: add a dataset field to events before they reach Cribl Search, in Cribl Stream or any upstream sender, and Search skips Dataset rules and routes directly to that Dataset; a missing Dataset sends the event to main.

Metric Dataset rules are the metrics counterpart. By default a single Metrics catch-all rule matching * sends all metrics to the primary metrics Dataset. They match on the internal __inputId field in sourceType:sourceId form, for example prometheus_rw:in_prometheus_rw, with specific rules above the catch-all. They differ from Log rules in two ways: no datatype matching, and Send data to offers only metrics Datasets, one per engine.

Datatype rules: shaping data

Datatype rules run earlier. Per the Datatype Rules page, logs flowing from Sources into a lakehouse engine are parsed into structured events through Datatyping, which applies to logs, not metrics.

The workflow has four steps. First, Auto-Datatyping: by default, Cribl Search automatically analyzes incoming log events and assigns a matching Datatype, needing no configuration. Second, check for uncategorized data: unmatched logs display as Uncategorized, and the Datatyping page can sample them live or search the past 24 hours. Third, add Datatype rules mapping specific patterns to stock v2 Datatypes. Fourth, add a custom v2 Datatype for the rest, and point a rule at it.

A Datatype rule takes a name, a KQL expression in Kusto expression to match aimed at uncategorized data, and a Datatype to assign, from stock v2 or custom v2 Datatypes. Expressions point at the fields guaranteed before Datatype assignment: _raw, the raw event text, and __inputId; the conventions match the rules above. The page's failed-login rule, copied from the shape-data page:

_raw = "*failed login*"

Cribl Search interprets each event's timestamp, using the event's own timezone information when present and assuming UTC otherwise. Fix offsets before ingestion with the _time override field, at Datatyping through a custom v2 Datatype's timestamp extraction settings, or at Dataset assignment with Modify fields.

Override fields bypass Search processing per event: _time skips timestamp parsing, using the given UTC value as the event time; datatype skips Datatype rules and applies the named stock or custom v2 Datatype, falling back to Auto-Datatyping if it doesn't exist; isParsed, always paired with datatype, uses the fields already on the event instead of reparsing _raw. For pre-parsed events from Cribl Stream or Cribl Edge, set datatype and isParsed together, plus dataset if you route upstream.

What the docs flag

Both the metrics Dataset and Metric Dataset rules are Preview features: Cribl is still developing them, doesn't recommend production use because they might not be fully tested or optimized, documentation could be incomplete, and support might be limited.

Metric Dataset rules apply as data arrives and aren't retroactive: metrics that arrive before a matching rule exists fall through to the catch-all and stay in the primary metrics Dataset. Retention drives storage planning; the docs estimate log compression between 10:1 and 12:1, and the single-Dataset storage formula assumes a 10:1 ratio.

One naming inconsistency: create-search-datasets records an invalid dataset value in main as invalid_dataset_provided, while organize-data and shape-data describe the same fallback as _dataset_reason set to "does not exist". This post reports each page's value as written.

Clearing a Search Dataset and deleting a Log Dataset are both one-way and require Maintainer access. Accelerated fields add ingest and index costs, hence the recommended maximum of 15.

Frequently asked questions

How do Log Dataset rules decide where events land in Cribl Search?

Log Dataset rules run top-down, and the first match wins, so specific rules go above broader ones. Events that match no rule, or a rule pointing at a deleted Dataset, fall back to the main Dataset.

What happens to logs with out-of-range timestamps in a Cribl Search Dataset?

Cribl Search retimestamps out-of-range logs, resetting _time to now() while preserving the original value in _original_time. Out-of-range metric samples are rejected instead of corrected. Default latest bound: one day after now() for logs, 10 minutes for metrics.

Can I skip Cribl Search Dataset rules by setting a field upstream?

Yes. Adding a dataset field to events before they reach Cribl Search makes it skip Dataset rules and route them directly to that Dataset. Events bound for a Dataset that does not exist land in the main Dataset.

How do I handle logs that Cribl Search leaves as Uncategorized?

You add a Datatype rule whose KQL expression matches the uncategorized subset and assign a stock v2 or custom v2 Datatype. For logs the stock Datatypes do not cover, add a custom v2 Datatype first, then point a rule at it.