Feeding Cribl Search: Lake, S3, and Edge

How Search gets data

Cribl Search reads data through Datasets. The three connector guides, for Cribl Lake, Amazon S3, and Cribl Edge, converge on one shape: a Cribl Search Dataset that names the data and pairs it with the Datatype or Datatypes that turn stored files into events, referenced by ID in your queries.

The paths get there differently. Amazon S3 is built from scratch: a Dataset Provider that, per the Amazon S3 guide, "tells Cribl Search where to query and contains access credentials", then a Dataset pointing at that provider. For Cribl Lake, Search already includes the Dataset Provider; the Dataset mirrors a Lake Dataset you create in the Lake UI. Cribl Edge ships with built-in Datasets you can search immediately.

From Cribl Lake

The Cribl Lake connector page starts with a short claim: Cribl Lake Datasets "work as Cribl Search Datasets out-of-the-box, so you can instantly start searching them." The provider needs no setup, because Search ships with a preconfigured one called cribl_lake, visible under Products > Search > Data > Dataset Providers. So the order is: create the Lake Dataset in the Cribl Lake UI, then add or select the matching Federated Dataset in Cribl Search, set its Dataset Provider to the Lake provider, and choose the Dataset Type.

Lake-backed Search uses Dataset Type v1 or v2, and v2 supports newline-delimited JSON, Parquet, and DDSS objects in Lake storage. Lake Datasets skip the Filters table that Amazon S3 and Azure Blob Storage Datasets use, because Search already knows how Lake lays out and names objects. On Type v2 a Datatypes table appears instead, with Search pre-filling the first row from the Lake Dataset's storage format: generic_ndjson for JSON, cribl_lake_parquet for Parquet, splunk_journal for Splunk DDSS. DDSS Datasets get a single row and no Add Data Formats button.

The speed control is the Lakehouse. With a Dataset assigned to one, queries whose time range falls within the configured Lakehouse retention period run "significantly faster"; data aged out of the Lakehouse but still in the Dataset runs at standard speed. A multi-Dataset query keeps Lakehouse speed only if every Dataset in it is Lakehouse-assigned and every operator is on the page's list, or centralize or limit comes before any other operator. You can turn the Lakehouse off for a Dataset. Copied from the Cribl Lake connector page:

set lakehouse="off" dataset="lakehouse_dataset"

The reverse direction, via the export operator, is on the Cribl Lake connector page:

// export Cribl Search results to an existing Lake Dataset
dataset="cribl_search_sample"
| export to myLakeDataset

One condition on that round trip: exports usually carry data Search has already parsed into fields, which keeps it Lakehouse compatible; unparsed data cannot be searched at Lakehouse speed.

From Amazon S3

The Amazon S3 connector page is the longest of the three, because it builds both halves from scratch. First a Dataset Provider: a unique ID starting with a letter, the type Amazon S3, and an Authentication method of Assume Role or AWS keys. AssumeRole requires the IAM role's AssumeRole ARN; session duration defaults to 3600 seconds, ranging from 900 to 43200. AWS keys requires the IAM user's Access key and Secret key. Advanced settings add Endpoint, Signature version (defaults to v4), and toggles for reusing connections and rejecting unverified certificates.

Then a Dataset, type Federated Dataset. Type is v1 (the default) or v2, with v2 Datasets supporting faster search and requiring NDJSON, Delimited Text, Parquet, or Splunk Journal format. A v2 Dataset sets a Bucket path, one or more Filters rows pairing a glob with a Datatype ID, a Partitioning scheme (default Defined in Path, or Splunk DDSS / Splunk SmartStore), and a Region toggle that, on by default, makes Cribl detect the Region and lock the Region drop-down. A Splunk scheme replaces the Filters table with one Datatype drop-down, limited to Splunk Journal Datatypes; Search then discovers the journal files from the index's parent folder.

A v1 Dataset defines at least one path, Region, and partitioning configuration, and Add bucket/path adds more. Each path can carry a Path filter, a JavaScript expression evaluated against the path that defaults to true. Paths support tokens like ${data}, which become event fields, and time tokens for time-partitioned data; Skip event time filter then switches the query's time range to partition boundaries instead of event timestamps.

Processing assigns the Datatypes that break data into events; Storage classes requires at least one selection, and warmer-only choices minimize retrieval costs.

DDSS and SmartStore layouts put the index above the time-based part of the path, so a time range alone leaves Search traversing every index. The doc's answer, copied from the Amazon S3 connector page:

dataset="my_ddss_dataset" earliest=-7d latest=-1d
| where index=="my_index"

Name the index with the where operator and case-sensitive ==, spelled exactly as it appears in the path; only an exact match narrows it.

Transfer is free if the bucket is in the same AWS Region as your Cribl.Cloud workspace. Otherwise AWS data transfer costs apply: you pay the transfer and API costs, they are not covered by Cribl.Cloud credits, and LIST and GET requests are additionally charged. Search will attempt to read S3 Standard, S3 Intelligent-Tiering (objects that have not aged into an unsupported class), S3 Express One Zone, S3 Standard-IA, S3 One Zone-IA, and S3 Glacier Instant Retrieval. S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive are unsupported, so Search skips those objects and reports their count in the skipped metric. To search an archived object, restore it in Amazon S3 first, then select Restored from Archive on the Storage classes tab.

From Cribl Edge

The Cribl Edge connector page starts from the lightest position: with Edge Nodes already running, you set up nothing in Search and start searching immediately. Edge collects logs, metrics, and application data and sends them to Cribl Stream or any other supported destination in real time, and Search queries the data on the Nodes. Its worked example installs an Edge Node on an Ubuntu EC2 instance.

Search ships with several built-in Cribl Edge Datasets that need no changes; for other logs in the filesystem Edge can read, you add a custom Dataset. Its Path is a JavaScript expression supporting environment variables, wildcards, and tokens; its example /foo/${country}/${state}/${city}/ makes the tokenized segments fields on the Dataset's events. A Path Filter evaluated against the path defaults to true and can narrow the search. Advanced Settings add a Partitioning scheme of Splunk DDSS or Splunk SmartStore, with Search discovering the content automatically when the bucket follows the default layout. Both layouts put the index above the time-based part of the path, so the doc shows the same index-naming pattern as the S3 guide. Copied from the Cribl Edge connector page:

dataset="my_ddss_dataset" earliest=-7d latest=-1d
| where index=="my_index"

Processing uses Datatypes, as elsewhere. Access is the piece to watch: Search users can search only the Edge Nodes they can access via Cribl Edge Permissions or Roles. Admins and Editors can create Datasets pointing at any Node, but at query execution time Search queries and returns results from only the Nodes the querying user can access.

What the docs flag

• Lakehouse speed is conditional. It holds only while the query's time range stays within the configured Lakehouse retention period, under those operator rules, and never for unparsed exports.
• S3 costs are on you. Beyond same-Region transfers, AWS data transfer and API costs are not covered by Cribl.Cloud credits, and LIST and GET requests are charged separately.
• Cold storage classes are skipped, not errored. S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive objects are unsupported and counted in the skipped metric.
• One bad path sinks the search. On a multi-path S3 Dataset, a failure on any path fails the whole search.
• Skip event time filter has a precondition. The S3 page warns against it unless the paths use time-based tokens.
• Edge visibility is per user. A Dataset may point at more Nodes than the querying user can see; results stay restricted to that user's access.

Frequently asked questions

Do I need to add a Dataset Provider for Cribl Lake in Cribl Search?

No, Cribl Search ships with a preconfigured Dataset Provider for Cribl Lake called cribl_lake. You create the Lake Dataset in the Cribl Lake UI first, then add or select the matching Federated Dataset in Search.

Which S3 storage classes does Cribl Search skip?

Cribl Search does not support S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive, skipping objects in those classes and counting them in the skipped metric. To search an archived object, restore it in Amazon S3 first and select Restored from Archive on the Storage classes tab.

When do Cribl Search queries run at Lakehouse speed?

When the Dataset is assigned to a Lakehouse and the query's time range falls within the configured Lakehouse retention period. Multi-Dataset queries also need every Dataset Lakehouse-assigned and the page's operator rules, or they run at standard speed.

Can every Cribl Search user query every Cribl Edge Node?

No. At query execution time, Cribl Search queries and returns results from only the Edge Nodes the querying user has access to via Cribl Edge Permissions or Roles, even if an Admin or Editor pointed the Dataset at any Node.