Splunk Cloud DDSS with Cribl Lake

What DDSS is

The Splunk Cloud Self Storage (DDSS) Direct Access page opens by promising to bring "Splunk data straight into Cribl Lake, bypassing Cribl Stream processing." It defines the option plainly: "You can use the Splunk Cloud Self Storage option to archive one or more Splunk Cloud indexes directly to Cribl Lake. This option bypasses routing or processing it through Cribl Stream. Once your data is integrated with Cribl Lake, Cribl Stream and Cribl Search can access it as a Lake Dataset."

The two pages this post draws from word the acronym differently. The About Cribl Lake page expands DDSS as "Splunk Dynamic Data Self Storage," while the Splunk Cloud Self Storage page title gives it as "Splunk Cloud Self Storage (DDSS)." I keep each page's wording as printed. What both pages share is the mechanism: Splunk Cloud indexes are archived into Cribl Lake directly, no routing or processing through Cribl Stream happens in between, and the archived data shows up as Lake Datasets.

The Splunk page sets one requirement before setup begins: you need a Splunk index, new or existing. It also states that you can configure one Splunk Cloud Direct Access Source per Workspace. Neither page discusses pricing or SLAs for this integration, so this post makes no claims about either.

How Lake delivers it

To configure the integration, the page says you work "back and forth between Splunk Cloud and Cribl Lake to share linking identifiers." The major steps, copied from the Splunk Cloud Self Storage (DDSS) page:

1. In Splunk Cloud, get your Splunk Cloud ID.
2. In Cribl Lake, create a Cribl Lake Source.
3. In Splunk Cloud, connect your Cribl Lake bucket.
4. In Cribl Lake, provision your Lake.
5. In Splunk Cloud, test and submit the connection.
6. In Cribl Lake, create a Cribl Lake Dataset.

The first identifier is the Splunk Cloud ID, which the page defines as "the first part of your Splunk Cloud URL and the 12-character S3 bucket string." Its example walks through it: if your Splunk Cloud URL is hungrygoatscomputing.splunkcloud.com, then your Splunk Cloud ID is hungrygoatscomputing-{12-character s3 string}. The page offers two ways to retrieve the AWS bucket prefix, presented in API and Splunk UI tabs: the Splunk Cloud Platform Admin Config Service (ACS) API, which manages DDSS self storage locations programmatically, or the Splunk Cloud Platform workflow that configures an Amazon S3 self storage location, where the required bucket name prefix appears in the Amazon S3 bucket name field.

Next, in Cribl Lake, select Direct Access from the sidebar, choose the Splunk Cloud Self Storage Source and select Next, paste in your Splunk Cloud ID, optionally enter a Description, pick the AWS Region of your Splunk Cloud Platform from the Region drop-down, and copy the resulting Cribl Lake bucket name to your clipboard. Back in Splunk Cloud, paste that bucket name in the Amazon S3 bucket name field, select Generate to generate a bucket policy, and copy it. Back in Cribl Lake, paste the policy into the AWS S3 bucket policy pane, select Save, and wait a few minutes for the new Lake to be provisioned. The page notes that "Cribl provisions the underlying object storage for you during this step. Cribl-managed Lake storage is encrypted at rest by default."

The connection is verified from the Splunk Cloud side. In the Self Storage Locations dialog, select Test; the page says "Splunk Cloud will write a 0 KB test file to verify that Splunk Cloud Platform has permissions to write to Cribl Lake." If you see a success message, select Submit to finish linking your Self Storage location to Cribl Lake, and if the test fails the page points to its troubleshooting section.

The final step creates the Lake Dataset. Each Splunk index maps to a single Lake Dataset, and the page says no additional Search dataset setup is required. It does list three specific requirements to handle DDSS data: give the Dataset an ID that exactly matches your Splunk index name, because otherwise you will be unable to search or replay your data; set the Storage Location to the default Cribl Lake, or make no selection on the drop-down; and set the Data format to DDSS. Save the Dataset and, per the page, your Splunk data should begin flowing into it.

Searching the archived data is where the naming choice pays off. Because the Dataset ID equals the index name, naming the Dataset in a query also names the index, and "Cribl Search scopes the search to that index for you, so you don't need an index filter in the query." The page's example query, copied from the Splunk Cloud Self Storage (DDSS) page:

dataset="my_index" earliest=-7d latest=-1d

To search several archived indexes at once, the page shows combining Datasets, as in dataset="my_index" or dataset="my_other_index", or matching them with a wildcard, as in dataset="my_*". It also contrasts this with a Cribl Search federated Dataset that points at a Splunk self-storage bucket of your own, where one Bucket path covers every index and you name the index in the query instead.

Where this fits in Lake overall

The About Cribl Lake page describes the product as "a data lake solution for long-term, full-fidelity storage of IT and security data" that receives data from various sources via Cribl Stream or Direct Access, and stores that data, "enabling you to later query the data in Cribl Search, or to replay it through Stream as needed." Direct Access is the intake path this integration belongs to. The page's list of ways to send data to the Lake includes the Cribl Lake Destination in Cribl Stream, the export operator for Cribl Search query results, and Direct Access (HTTP) for senders such as FluentBit, Logstash, Vector, and the OpenTelemetry Collector, and it lists the Splunk Cloud Self Storage option as the dedicated way "To archive Splunk Dynamic Data Self Storage (DDSS) data to Cribl Lake." On the storage side, the page says you can store your data in gzip-compressed JSON, Parquet, or DDSS depending on use case, that Lake Datasets work as Cribl Search Datasets out-of-the-box, and that Lake adds a dedicated Collector Source to Cribl Stream for replaying data you have stored.

What the docs flag

The Splunk page keeps a short limitations list, each item scoped to a Lake Dataset created or populated via Splunk Cloud Direct Access:

• Such a Dataset "is read-only to other Cribl products." Both Cribl Stream and Cribl Search can process data from the Dataset, but neither product can write to it.
• A Dataset populated via Direct Access "must use the default Cribl Lake Storage Location, not an external bucket."
• A Dataset created via Splunk Cloud Direct Access "can't be assigned to a Lakehouse," while the About page lists Lakehouse assignment as a general Lake feature for Datasets.
• For Cribl-managed storage, retention is enforced at the Dataset level only.

Linked resources also mean deletion on either side has stated consequences. Deleting the Splunk Cloud Self Storage Source in Cribl Lake means you lose all the Splunk data archived through it and Splunk Cloud can no longer write to Cribl Lake through the integration, though the deletion does not affect data flow and storage within Splunk Cloud. Deleting a Self Storage location in Splunk Cloud that is linked to a Cribl Lake Dataset means Splunk Cloud stops archiving data to Lake, but data already stored in Cribl Lake is unaffected. If you cancel your Splunk Cloud account, the page notes the Source in Cribl Lake can still remain intact, letting Cribl Stream (Collector) and Cribl Search (queries) keep reading the Dataset's current contents, with no new data able to be written.

The troubleshooting section is narrow but blunt. If Splunk Cloud fails to write the test file to your Lake Dataset, the page lists regions mismatched between Splunk Cloud and Cribl.Cloud, a bucket name that does not start with the Splunk Cloud ID, and an Amazon S3 bucket policy not pasted in accurately, and it states that correcting each of these requires re-creating your integration from scratch.

Frequently asked questions

Can Cribl Stream write to a Lake Dataset populated via Splunk Cloud Direct Access?

No. A Lake Dataset created and populated via Splunk Cloud Direct Access is read-only to other Cribl products, so Cribl Stream and Cribl Search can process data from it but neither can write to it.

Where does Splunk Cloud Self Storage data land in Cribl Lake?

In a Cribl Lake Dataset, with one Dataset per Splunk index. The data is archived directly into Lake, bypassing Cribl Stream routing or processing, and Cribl provisions the underlying object storage during setup, with Cribl-managed Lake storage encrypted at rest by default.

What happens when you delete a Splunk Cloud Self Storage Source in Cribl Lake?

You lose all the Splunk data archived through that Source, and Splunk Cloud can no longer write to Cribl Lake through that integration. The deletion does not affect data flow or storage within Splunk Cloud.

Why must a DDSS Lake Dataset ID match the Splunk index name?

Because naming the Dataset in a query also names the index, and Cribl Search scopes the search to that index for you. If the ID does not match exactly, the page warns you will be unable to search or replay your data.