Cribl Lake 101: The Cold Storage Layer for Log Data

What Cribl Lake is

According to the About Cribl Lake page, Cribl Lake is "a data lake solution for long-term, full-fidelity storage of IT and security data." It receives data from various sources via Cribl Stream or Direct Access and stores that data, "enabling you to later query the data in Cribl Search, or to replay it through Stream as needed." Besides the data you explicitly send, Lake "also receives and retains internal logs and metrics from the Leader of your Cribl.Cloud Organization."

What the page lists among the features Cribl Lake provides:

• Create a data lake in a few clicks and a few minutes.
• Store any type of IT and security data: raw, structured, or unstructured.
• Set and manage retention and access-control policies through a graphical UI, "reducing dependencies among multiple teams."
• Use Cribl Stream Pipelines and Functions to control the mix, volume, and structure of data sent to your lake.
• Search your Lake data in place, using Cribl Search, "to minimize ingress charges for analysis."
• Assign Lake Datasets to Lakehouses "for dramatically faster search responses and more-predictable billing."

Cribl Lake is available only in Cribl.Cloud, and it can connect, via a Collector or a Destination, with both Cribl-managed Cloud and customer-managed hybrid Stream Worker Groups, with the hybrid groups required to run version 4.8 or higher.

The storage model is worth understanding before you go further. Cribl Lake organizes your data into Datasets, and the data "is stored in open, non-proprietary formats, which avoids vendor lock-in." You do not have to think about partitioning configuration: "Cribl Lake takes care of the schema and organizes the data for you." Depending on your use case, data can live in gzip-compressed JSON, Parquet, or DDSS. On a Cribl.Cloud free plan you can store up to 50 GB of data, and data in the cribl_metrics and cribl_logs Datasets is stored for 30 days free of charge.

The page lists other paths into the Lake: results of a Cribl Search query via the export operator, Splunk DDSS data via the Lake Splunk Cloud Self Storage option, and data over HTTP through Direct Access, which supports FluentBit, Logstash, Vector, the OpenTelemetry Collector, and other senders. On the way out, Lake adds a dedicated Collector Source to Cribl Stream that replays stored data via Routes or QuickConnect, and Lake Datasets work as Cribl Search Datasets out-of-the-box, so you do not create additional Dataset Providers or Datasets in Search.

Datasets: the unit of storage

The Manage Lake Datasets page focuses on the Datasets you populate from Cribl Stream using the Cribl Lake Destination, though selected types of data can also be created through Direct Access, and Cribl Search results can be sent in with the export operator.

To create a new Dataset, select Datasets in the Cribl Lake sidebar, then Add Dataset, and enter an ID. The identifier can contain letters (lowercase or uppercase), numbers, and underscores, up to 512 characters. It must be unique and must not match any existing Cribl Search Dataset, because "you can't change the identifier after a Dataset is created, so make sure it is meaningful and relevant to the data you plan to store in it." The form, rearranged from the page:

Add Dataset
  ID                        letters, numbers, underscores; up to 512 characters; cannot change later
  Description               optional
  Storage Location          internal Cribl Lake storage (default) or an external Storage Location
  Storage Class             S3: Standard (default) | Azure: Inferred (default); S3- or Azure-backed only
  Retention period          anywhere from 1 day to 10 years
  Data format               JSON (default) | Parquet | DDSS (for Splunk Cloud Self Storage)
  Lakehouse                 optional; linking at creation makes the Dataset a mirrored Dataset
  Advanced Settings         up to three Lake Partitions; up to five Lakehouse Indexed fields when linked

Once saved, you select the Dataset by its identifier in a Cribl Lake Destination or Collector. How many Datasets you may have depends on the Storage Location: 999 for Cribl-managed Lake, Amazon S3, Splunk Cloud Self Storage (DDSS), and Netskope Direct Access; 99 on an Azure Storage Location where each Dataset carries its own retention, or 1000 with a default retention period set on the Azure location.

You can edit a Dataset's description, Storage Location, retention period, partitions, linked Lakehouse, and Storage Class, but not its identifier. Changing retention affects all data in the Dataset: an increase makes current data adopt the new period, while decreasing it means "data older than the new time window will be lost."

Deletion is not instant. A selected Dataset is marked Deletion in progress, after which you can no longer edit it or connect it to Collectors or Destinations; data older than 24 hours is removed the following day, after midnight UTC, and the Dataset is only deleted once all of its data is 24 hours old or older. Built-in Datasets, and any Dataset with connected Collectors or Destinations, cannot be deleted.

Each Dataset can carry up to three partitions, and, with a Lakehouse, up to five Lakehouse-indexed fields, to speed up searching and replaying; a typical use case is accelerating hostname or sourcetype across many hosts or sources. Order matters: "make sure you provide broader partitions or Lakehouse-indexed fields first." The page steers you away from fields that contain PII, contain objects, or start with an underscore (such as _raw), and, for partitions, additionally away from high-cardinality and nullable fields.

On format, the form says to keep the default JSON to prioritize storage efficiency and to choose Parquet to optimize for search performance. JSON fits getting-started, semi-structured data, high compression with ordinary search performance, and storing original events in _raw for full-event retrieval. Parquet fits structured data with multiple fields, columnar storage, and queries that filter and aggregate on specific fields; it is not recommended for unstructured events such as CSV values, where "you will see no performance improvement over JSON." DDSS is for archiving data from Splunk Cloud using Self Storage.

Sending data in from Cribl Stream

The Cribl Lake Destination is described on the Lake docs as "a Cribl Stream feature that optimizes data management for two other Cribl products: It delivers data to Cribl Lake, and automatically selects a partitioning scheme that works well with Cribl Search." Its type is Non-Streaming, TLS support is yes, and PQ support is no.

It receives data from both Cribl-managed Cloud and customer-managed hybrid Stream Worker Groups, with hybrid groups required to run Cribl Stream version 4.8 or later. On hybrid hosts it "requires outbound HTTP/S access to port 443," and it does not allow selecting a different port. Storing in a regular Lake Dataset needs no schema, but to make full use of a Lakehouse, "you need to make sure that events sent from Cribl Stream are parsed into distinct fields before sending."

To configure it, open Stream under Products, choose a Worker Group, and open the New Destination modal via QuickConnect or the Routing UI. The settings, as the page shows them:

General Settings
  Output ID             unique name to identify this Cribl Lake Destination
  Lake dataset          a Lake Dataset to send data to (built-in cribl_logs and
                        cribl_metrics cannot be targeted with this Destination)

Optional Settings
  Backpressure behavior Block (default) or drop events when all receivers are
                        exerting backpressure
  Tags                  names to filter and group Destinations; a tab or hard
                        return between names; not added to processed events

Post-Processing
  Pipeline              Pipeline or Pack to process data before sending
  System fields         cribl_pipe included by default; supports c* wildcards;
                        other options: cribl_host, cribl_input, cribl_output,
                        cribl_route, cribl_wp

After configuring, select Save, then Commit & Deploy, and verify that data is searchable in Cribl Lake.

One mechanism the page spells out is _time handling: "The Cribl Lake Destination requires _time to be a number and not null. If an incoming _time value is null or a string, this Destination will convert the value to Date.now() / 1000." That conversion "creates a time-based partitioning scheme that is readily searchable."

For hybrid Worker Groups sitting behind a firewall or proxy that performs SSL inspection, the page's troubleshooting path is to open Organization Details, note the Organization ID and Region, and swap both into this FQDN format so your security team can add an exception:

lake-main-<organizationId>.s3.<region>.amazonaws.com
lake-main-goaty-mcgoatface.s3.us-west-1.amazonaws.com

Then verify outbound port 443 is open and test the Destination end to end.

Frequently asked questions

What is a Cribl Lake Dataset?

A Cribl Lake Dataset is the unit of storage in Cribl Lake, identified by a unique ID of up to 512 characters that you choose. It defines a Storage Location, a retention period from 1 day to 10 years, and a data format of JSON, Parquet, or DDSS.

How do I send data from Cribl Stream to Cribl Lake?

You create a Cribl Lake Destination in Cribl Stream, set an Output ID and the target Lake dataset under General Settings, then Save and Commit and Deploy. The Destination is Non-Streaming, supports TLS, and automatically selects a partitioning scheme that works well with Cribl Search.

What storage formats can Cribl Lake hold?

Cribl Lake stores data in open, non-proprietary formats, and a Dataset's format can be set to JSON, Parquet, or DDSS. The form keeps JSON as the default to prioritize storage efficiency, Parquet optimizes performance for search, and DDSS is for data archived via Splunk Cloud Self Storage.

Is Cribl Lake available only in Cribl.Cloud?

Yes, Cribl Lake is available only in Cribl.Cloud, where a free plan stores up to 50 GB of data. It connects through a Collector or Destination with both Cribl-managed Cloud and customer-managed hybrid Stream Worker Groups, and hybrid Worker Groups must run version 4.8 or higher.