Log Analysis with the Lakehouse

The lakehouse idea for logs

The Lakehouse logs overview in the Cribl Lake docs opens with one line of intent: use a Cribl Lakehouse to efficiently store, and quickly analyze, high-volume recent telemetry data. The overview frames that promise around a concrete workflow: ingest log files into a Lakehouse, then use Cribl Search to periodically analyze only the specific data relevant for threat-hunting or security investigations. Especially as data volume increases, the page presents the pattern as a cost-effective alternative to bulk-forwarding data to a traditional SIEM system.

The demo walks a sequential setup in the upstream logs sender, Cribl Stream, Cribl Lake, and Cribl Search, and reduces the pipeline to these broad steps:

1. Collect and stage logs in the applications and frameworks that generate them.
2. Shape the data, using a Cribl Stream Parser.
3. Create a Cribl Lake Dataset to store your parsed data.
4. Route the parsed data to the new Lake Dataset.
5. Configure a Lakehouse, to support low-latency analytics in Cribl Search.
6. Analyze relevant data, using Cribl Search ad hoc and scheduled queries.
7. Visualize your data, using Cribl Search predefined or custom Dashboards.

In the worked examples the shape stays consistent. Log files are written from their Amazon services into an S3 bucket and picked up by a Cribl Stream Amazon S3 Source. A Pipeline with a single Parser Function extracts the named fields that the Lakehouse requires, and a Cribl Lake Destination routes the parsed data into the new Dataset, reachable through either QuickConnect or Data Routes. With the flow in place, you add a Lakehouse sized to match your expected daily ingest volume and, once it is provisioned, assign the Lake Dataset to it.

Analyzing log data

The overview demonstrates three common Amazon log formats end to end. VPC Flow Logs are shaped with the AWS VPC Flow Logs parser library, which automatically sets the Type to Extended Log File Format. CloudTrail logs reuse that same library, because, as the page notes, it can handle CloudTrail logs, too; CloudFront logs use the AWS CloudFront library, which sets the Type to Delimited values. The page is explicit that these are examples: a similar workflow stores and analyzes many other types of logs, by substituting a different Stream Parser or configuring your own.

For analysis, the page names Cribl Search as the engine, with KQL as its query language. Its guidance for teams moving high-volume log analysis from a traditional SIEM to Cribl is that existing queries might need to be adapted to KQL. A built-in cribl_search_sample Dataset in Cribl Search contains VPC Flow Logs events, so you can experiment with queries before you enable continuous data flow and analysis. Cribl Copilot is named as well: Cribl AI can suggest KQL queries from natural-language prompts, and it can translate searches that already live in another system of analysis.

The first sample query retrieves rejected connections grouped by source address; it is copied from the Lakehouse logs overview:

dataset="<your_lakehouse_dataset>"
| action="REJECT"
| summarize by srcaddr

The second sample is one you could schedule to generate an hourly report that tracks traffic by source and destination, also copied from the Lakehouse logs overview:

dataset ="<your_lakehouse_dataset>" srcaddr="*" dstaddr="*"
| summarize count() by srcaddr, dstaddr
| sort by count desc

Searches can be scheduled to run on defined intervals, and once a query is scheduled you can configure Search Notifications as alerts that trigger on custom conditions you define. For visualization, the page points to Cribl Search Dashboards and to the Cribl Search AWS VPC Flow Logs Pack, whose sample AWS VPC Flow Logs Analysis Dashboard displays traffic volume, traffic accepted and rejected in both time-series and cumulative views, traffic by protocol, and top destination ports. Cribl AI can also suggest visualizations for a Dataset and build them from natural-language prompts.

Searching inside Lake

The Search Cribl Lake page covers the search side in its own right. Cribl Lake appears in Cribl Search as a preconfigured Dataset Provider called cribl_lake, which lets you use Lake Datasets as Cribl Search Datasets and start searching them right away: from your Organization's top bar, select Products, then Search, and your Lake Datasets are queryable from there. For Lake-backed federated Datasets on Dataset Type v1 or v2, including filter rows on Type v2, the page defers to the Connect Cribl Search to Cribl Lake setup guide.

Caching is the speed mechanism. The page says you can speed up searching your Cribl Lake data by caching it in a Lakehouse, and once a Lake Dataset is linked to a Lakehouse, searches against that Dataset run significantly faster, as long as the query's time range falls within the configured Lakehouse retention period. The tracking bar on search results lets you verify whether a search used a Lakehouse; if it did not, the bar presents potential reasons, with the examples given as a disabled or misconfigured Lakehouse, or a query exceeding the time range of the data stored in the Lakehouse.

A single query can run against multiple Lakehouse Datasets, and the page states the conditions for keeping lakehouse speed. All Datasets in the query must be Lakehouse-assigned, and either every operator comes from the allowed group (listed as cribl, centralize, extend, extract, foldkeys, limit, mv-expand, pivot, project, project-away, project-rename, search, and where), or, when any other operators are used, a centralize or limit operator is inserted before them, in which case result counts are further constrained by usage group limits. If neither condition is met, or the query includes non-Lakehouse Datasets, the query runs at standard speed.

The page's first example specifies the Dataset and limits the number of results; copied from the Search Cribl Lake page:

dataset="test_dataset"
| limit 100

The second example shows a partition speeding up retrieval: a search over the partitioned Dataset, which has a Lake partition named sourcetype configured for it, also copied from the Search Cribl Lake page:

dataset="partitioned" host="cribl-stream"

The last pair of examples turns the direction around: the export operator writes Cribl Search results to a Lake Dataset, and the efficient way to search those results is to provide the search job ID (found in the search details after running, or in the Search ID column of the History tab) to a where filter, copied from the Search Cribl Lake page:

dataset="exported_data"
| where source contains "1713177481843.9AOqxI"

You can also label exported events during export: the example adds the searching user to each record with extend, then searches the exported Dataset by that user. Both queries, copied from the Search Cribl Lake page:

dataset="cribl_search_sample"
| extend user = user()
| export to lake exported_data

dataset="exported_data"
| where user == "John Doe"

What the docs flag

• Lakehouse deprecation. The lakehouse-logs page leads with a deprecation notice: for lakehouse workloads, Cribl now recommends Cribl Search lakehouse engines, and Lakehouses in Cribl Lake will be deprecated in a future release. Existing Lakehouses continue to work with no immediate action required, Cribl will provide advance notice before you need to make changes, and your Cribl Account Team can help plan a migration from Lakehouse to a lakehouse engine.
• Speed is condition-bound. The faster-search claim attaches to the Lakehouse retention period, and multi-Dataset speed attaches to both the Lakehouse-assigned rule and the operator rules; miss any of them and the query runs at standard speed. The tracking bar is how you find out a search fell back.
• Behavior differs with caching. Executing Cribl Search queries against a Lakehouse-assigned Dataset changes some behavior and results, compared to executing the same queries without Lakehouse caching. The search page defers the details to the Cribl Lake/Lakehouse Search Differences page rather than restating them.

Frequently asked questions

Does a Cribl Lake lakehouse analyze log data on its own?

No. The overview routes analysis through Cribl Search, which defines ad hoc and scheduled KQL queries against the Lake Dataset assigned to the Lakehouse. In this flow the Lakehouse stores the recent, high-volume data and supports low-latency analytics for those queries.

Which Amazon log formats does the Cribl Lake lakehouse overview demonstrate?

It demonstrates VPC Flow Logs, CloudTrail logs, and CloudFront logs, each shaped by a Cribl Stream Parser before storage in a Lake Dataset. The CloudTrail example reuses the AWS VPC Flow Logs parser library, while CloudFront uses the AWS CloudFront library with the Delimited values type.

When does a Cribl Search query against Cribl Lake run at Lakehouse speed?

When all Datasets in the query are Lakehouse-assigned, the query's time range falls within the Lakehouse retention period, and the operator rules are met, either all operators from the allowed group or a centralize or limit operator before any others. If any condition is missing, the query runs at standard speed.

What do the Cribl Lake docs say about the future of Lakehouses?

Lakehouses in Cribl Lake will be deprecated in a future release, and Cribl now recommends Cribl Search lakehouse engines for lakehouse workloads. Existing Lakehouses continue to work, with advance notice before any required changes, and the Cribl Account Team can help plan a migration to a lakehouse engine.