Lake Datasets and Storage: S3, Azure, and More

Datasets and their settings

A Lake Dataset is the named unit Cribl Lake uses to hold data. The Manage Lake Datasets page documents creation, editing, deletion, partitions, formats, and storage classes. Creation: Datasets > Add Dataset, then enter an ID (letters, numbers, underscores; up to 512 characters) plus an optional Description. The ID can't be changed after creation, must be unique, and must not match any existing Cribl Search Dataset.

The Storage Location drop-down always includes the internal Cribl Lake option, the default when no external Storage Location is configured. Retention runs from 1 day to 10 years; on an Azure location with a default retention period, the Dataset inherits it and can't be changed. The default Data format is JSON; Parquet optimizes for search performance; DDSS is available for Datasets populated via Splunk Cloud Self Storage.

Dataset creation caps: 999 for Cribl-managed Lake, S3, DDSS, and Netskope Direct Access; on Azure, 99 when Datasets carry per-Dataset retention, or 1000 with a location default retention. A Dataset can link to a Lakehouse (becoming mirrored) and take up to three Lake Partitions and five Lakehouse Indexed fields. Lakehouses can't be linked to Datasets populated via Splunk Cloud Direct Access or those using an external Storage Location.

Editing covers description, Storage Location, retention, partitions, linked Lakehouse, and Storage Class (S3- and Azure-backed). Raising retention is picked up by existing data; lowering it loses older data. Deletion is scheduled: the Dataset is marked Deletion in progress, data older than 24 hours is removed the following day after midnight UTC, and billing for it stops once marked. Built-in Datasets and Datasets with connected Collectors or Destinations can't be deleted.

Partitions and indexed fields speed up search and replay; the page's typical use case is hostname or sourcetype. Avoid fields with PII, objects, or a leading underscore (such as _raw), and, for partitions, high-cardinality or nullable fields; source and dataset are reserved. Format guidance: JSON for semi-structured or _raw-only data; Parquet for structured multi-field events; DDSS for Splunk Cloud archives. The Storage Class dropdown appears only for S3- or Azure-backed Datasets, and changes apply to newly written objects only.

Amazon S3

The Configure an Amazon S3 Storage Location page is the BYOS path for AWS: an S3 bucket keeps direct ownership of your data for regional or compliance purposes, and Cribl Lake keeps provisioning, retention, access control, and analytics. AWS charges apply in addition to Cribl Lake credits, and each location maps to one bucket.

Select Lake > Storage Locations > New Storage Location, provider AWS. Settings takes a name, optional Description, AWS Region, Bucket Name (unique on AWS), Encryption (default SSE-S3, selectable as SSE-KMS), and an External ID (random UUID recommended). Authentication is a CloudFormation flow: download the template, create the stack in the AWS console, and paste the stack's AccessRoleARN from Outputs into AssumeRole ARN. The Dataset's retention shows up in the S3 Console's Management tab, so don't change retention there, and don't point a location at an existing bucket that holds other data, since Cribl Lake needs full control of lifecycle rules and S3 Inventory.

The Storage Class dropdown on an S3-backed Dataset defaults to Standard and can also write to Intelligent-Tiering, Standard-IA, One Zone-IA, Glacier Instant Retrieval, Glacier Flexible Retrieval, and Glacier Deep Archive; Cribl Search reads the first five on demand, and the deeper Glacier classes are skipped until manually restored through the AWS S3 Console or API.

Supported regions are a fixed list: us-east-1, us-east-2, us-west-2, ca-central-1, eu-central-1, eu-west-1, eu-west-2, eu-west-3, eu-central-2, ap-southeast-1, ap-southeast-2, ap-northeast-1, sa-east-1. Cribl.Cloud Gov (FedRAMP Moderate) supports only us-east-1, us-east-2, and us-west-2, and no GovCloud regions for BYOS. To detach, remove connected Datasets, select Detach, type DETACH, and revoke Cribl Lake permissions in AWS.

The storageLocations REST API creates locations without a CloudFormation template, so the bucket and IAM role are yours to build. The creation endpoint, copied from the Amazon S3 API section:

POST /products/lake/lakes/{lakeId}/storage-locations

Attaching a Dataset via API references the Storage Location rather than a bucket name; the docs example, also from the Amazon S3 API section, sets the storage class explicitly:

datasets:
              - id: my_dataset
                storageLocation: my-byos-location
                retention: 30d
                format: json
                storageClass: STANDARD

Azure

The Configure an Azure Storage Location page uses a Blob Storage container as the Storage Location. The generated ARM template creates the storage account and assigns Cribl Lake's roles, so no advance setup is required. You do need a resource group (the account is created in its region, which can't be changed later), an Azure role that allows admin consent, and permission to deploy the template.

Select Lake > Storage Locations > New Storage Location, provider Azure. Settings takes a Storage Location ID and optional Description, then Tenant ID, Subscription ID, Resource Group Name, Storage Account Name, Region, and an optional Default retention period; enabling it makes new Datasets inherit location-wide retention (up to 10 years). Select Authorize Cribl.Cloud to consent, Generate ARM Template to download the JSON, and deploy it; the CLI form, copied from the Azure page:

az deployment group create \
              --resource-group <your-resource-group> \
              --template-file CriblLakeByosAzureArmTemplate.json

Confirm the deployment, select Save, and Cribl validates the connection, showing any missing permissions. Any publicly available Azure region works (the account's region is validated at save), and FedRAMP deployments can't use Azure storage locations.

Access tiers default to Inferred; the rest are Hot, Cool (held at least 30 days), Cold (at least 90 days), and Archive, which can't be searched on demand until rehydrated. Via the API, storageClass takes those values (default Inferred), and retention is omitted when the location sets a default; the docs example, copied from the Azure page:

datasets:
              - id: my_dataset
                storageLocation: my-azure-byos-location
                retention: 30d
                format: json
                storageClass: Cool

Splunk Cloud storage

The Splunk Cloud Self Storage (DDSS) Direct Access page archives Splunk Cloud indexes directly into Cribl Lake, bypassing Cribl Stream; Stream and Search then read that as Lake Datasets. One source is allowed per Workspace.

In Splunk Cloud, get your Splunk Cloud ID (the first part of your URL plus the 12-character S3 bucket string). In Cribl Lake, create the Splunk Cloud Self Storage Source with that ID and your Splunk Cloud Platform's AWS Region, and copy the resulting Cribl bucket name. Paste that into Splunk's Amazon S3 bucket name field, Generate the bucket policy, and paste it into Cribl Lake's AWS S3 bucket policy pane; Cribl provisions the object storage, encrypted at rest by default.

Select Test in Self Storage Locations; Splunk Cloud writes a 0 KB test file, and on success, Submit. A failed test (region mismatch, bucket name not starting with the Splunk Cloud ID, or incomplete policy paste) means re-creating the integration. Finally, create the Dataset: an ID exactly matching the index name, the default Cribl Lake Storage Location, and format DDSS; each index maps to one Dataset.

Because the Dataset ID equals the index, Cribl Search scopes the query to it without an index filter; the example, copied from the Splunk Cloud page:

dataset="my_index" earliest=-7d latest=-1d

Indexes can be combined with or or a wildcard. Direct Access Datasets are read-only to other Cribl products, can't be assigned to a Lakehouse, and have retention enforced at the Dataset level only.

Choosing a backend

• Ownership: S3 and Azure keep your data in your own account for regional or compliance reasons; DDSS inverts this, with Cribl provisioning the storage.
• Charges: only the S3 page states AWS charges in addition to Cribl Lake credits.
• Regions: S3 is a fixed list (FedRAMP Moderate: three regions); Azure accepts any public region, validated at save; FedRAMP excludes Azure.
• Capacity: 999 per Cribl-managed, S3, and DDSS location; 99 or 1000 on Azure, which alone has a location-level retention toggle.
• Cold tiers: S3 skips Glacier Flexible Retrieval and Deep Archive at search; Azure skips Archive until rehydrated; S3 adds Intelligent-Tiering, and Azure's Cool and Cold require 30 and 90 days minimum.

Frequently asked questions

Where can a Cribl Lake Dataset store its data?

A Cribl Lake Dataset stores its data in the internal Cribl Lake option, an Amazon S3-backed Storage Location, or an Azure-backed Storage Location; DDSS Datasets populated via Direct Access use the default Cribl Lake location. Each S3-backed Storage Location maps to one bucket.

How many Cribl Lake Datasets can I create on a storage location?

Cribl-managed Lake, Amazon S3, and DDSS locations each allow 999 Datasets, and an Azure location allows 99 with per-Dataset retention or up to 1000 with a location default retention that Datasets inherit. Azure is the only one of these with a location-level retention toggle.

Can I search Cribl Lake data in cold S3 classes or the Azure Archive tier on demand?

No. Cribl Search skips S3 objects in Glacier Flexible Retrieval or Glacier Deep Archive, and Azure Archive objects, until restored or rehydrated; S3 objects can be restored manually through the AWS S3 Console or API.

What restrictions apply to a Cribl Lake Dataset populated from Splunk Cloud Direct Access?

The Dataset must use the default Cribl Lake Storage Location, the DDSS format, and an ID matching the Splunk index name exactly. It is read-only to other Cribl products, can't be assigned to a Lakehouse, and has retention enforced at the Dataset level only.