Structuring Events for Cribl Lake

Cribl Lake stores events as gzip-compressed JSON or Parquet files, and the Structure Events for Cribl Lake page frames the workflow around that choice: preprocess your events in Cribl Stream before sending them to Lake, optimizing either for long-term retention or frequent access. The format you choose when creating a Lake Dataset, and how you structure your events in Cribl Stream, determine which optimization you get. The Cribl Lake Collector page covers the reverse direction.

Why structure matters in Lake

For data stored primarily for compliance or long-term retention, JSON is typically the better choice: JSON with gzip compression produces smaller files when your goal is to minimize storage costs and you don't need to query the data frequently. If you plan to search and analyze the data regularly, building dashboards, running investigations, or performing ad-hoc queries in Cribl Search, Parquet is often the better option: its columnar format enables faster queries, especially when events contain structured fields that Cribl Search can filter efficiently.

The page's highlights: format events in Cribl Stream before sending them to Cribl Lake; for maximum storage efficiency, use JSON and remove all fields except _raw; for best search performance, use Parquet and keep all fields.

The page's comparison table weighs four considerations, JSON against Parquet:

• Use case. JSON fits compliance, archival, and infrequent access; Parquet fits frequent search, dashboards, and analysis.
• Event structure. JSON requires only _raw; Parquet requires multiple parsed fields.

Structuring the data

For JSON, the recipe is to simplify events to just the original log event in the _raw field. A Pipeline in Cribl Stream removes all top-level fields except _raw, preserving the original event for compliance or archival use cases.

The page's example starts with a firewall log arriving in Cribl Stream with multiple extracted fields, copied from the Structure Events for Cribl Lake page:

{
  "_raw": "Jun 12 14:23:01 fw01 %ASA-6-302013: Built outbound TCP connection 847263 for outside:203.0.113.50/443 to inside:192.168.1.100/52344",
  "_time": 1718194981,
  "action": "Built",
  "protocol": "TCP",
  "src_ip": "192.168.1.100",
  "dest_ip": "203.0.113.50",
  "dest_port": "443"
}

Aiming for maximum storage efficiency, the Pipeline removes all non-essential fields, leaving only _raw and _time; the result, copied from the Structure Events for Cribl Lake page:

{
  "_raw": "Jun 12 14:23:01 fw01 %ASA-6-302013: Built outbound TCP connection 847263 for outside:203.0.113.50/443 to inside:192.168.1.100/52344",
  "_time": 1718194981
}

You can also keep other top-level fields to improve search performance later, at the cost of larger file sizes in Cribl Lake. The original log stays intact in _raw for compliance, and if you later need to replay this data for analysis, the page says a Cribl Lake Collector can re-parse it through Cribl Stream.

If _raw contains valid JSON, Cribl Search can extract fields from that JSON. The page notes this does not change the guidance: storing JSON objects in _raw is not an efficient way to get structured fields. For best search performance, send structured top-level fields and use Parquet.

Parquet shines when your events have multiple fields, particularly fields with repeated values, so fields like Kubernetes labels that appear across thousands of events are stored only once per file. For those benefits, your events must arrive at Lake as parsed, structured data with multiple top-level fields. Cribl Search also uses the structure through predicate pushdown: min/max values within the files let it skip irrelevant data during queries, significantly reducing query times when filtering on fields like namespace, pod_name, or log_level.

The page's Kubernetes example is copied from the Structure Events for Cribl Lake page. Keep all of these fields intact when sending to a Parquet Dataset:

{
  "_raw": "2024-06-12T14:23:01.847Z INFO [main] Application started successfully",
  "_time": 1718194981,
  "kube_namespace": "production",
  "kube_pod": "api-server-7d4b8c6f9-x2vnm",
  "kube_container": "api-server",
  "kube_node": "node-pool-1-abc123",
  "log_level": "INFO",
  "message": "Application started successfully"
}

The repeated values in kube_namespace, kube_node, and log_level compress efficiently, and Cribl Search filters directly on any of them without re-parsing the full _raw field.

The Lake Collector

The Cribl Lake Collector page describes the opposite direction: sending your data from Cribl Lake to Cribl Stream.

The collector can gather data from both Cribl-managed Cloud and customer-managed hybrid Stream Worker Groups; hybrid groups must be running version 4.8 or higher.

Configuration: select Products, then Stream, on the top bar; select a Worker Group under Worker Groups, then Data, then Sources on its submenu; select Cribl Lake under Collectors in the Sources tiles; Add Collector opens the New Collector modal.

The Collector Settings in the modal:

• Collector ID: the unique ID for this Collector (the page's example: myLakeCollector).
• Storage Location: where the Dataset stores its data; the drop-down lists Cribl Lake (default) and each external Storage Location your user can access, and the Lake Dataset list then shows only Datasets that use that location.
• Lake Dataset: the Dataset to collect data from, filtered by the selected Storage Location.
• Tags: optional tags for filtering and grouping Sources in Cribl Stream's Manage Sources page; not added to processed events, with (arbitrary) tag names separated by a tab or hard return.

You then select Save, then Commit & Deploy.

Result Settings determine how Cribl Stream transforms and routes the collected data. Event Breakers convert data streams to discrete events with rulesets applied, in order, to the input data stream (default System Default Rule), plus a buffer timeout (minimum 10 ms, default 10000 (10 sec), maximum 43200000 (12 hours)) for how long the breaker waits for new data before flushing the stream to the Routes as-is. Fields adds fields to each event, using Eval-like functionality: a Name and a JavaScript expression (which can be a constant) that computes the value. Send to Routes is on by default; off selects a specific Pipeline and Destination combination, on exposes an optional Pre-processing Pipeline, and the always-exposed Throttling field sets a rate in bytes per second or multiple-byte units (the page's example: 42 MB), with the default 0 indicating no throttling.

Advanced Settings cover post-processing and administrative options: Environment limits a GitOps-enabled config to one Git branch (empty means enabled everywhere), Time to live (default 4h) sets how long job artifacts stay on disk after completion and how long a job stays listed in Job Inspector, Remove Discover fields hides sensitive fields from Discover results (wildcards such as aws* work), and Resume job on boot resumes ad hoc collection jobs if Cribl Stream restarts mid-job.

When you run or schedule a collector job, you can set a Filter expression that Cribl Stream evaluates against both the Lake object path and the collected event fields. The page's warning about source: filtering on it restricts which Lake files Cribl Stream pulls, not only which events it keeps. Lake files are typically stored with extensions such as .json.gz, so a filter like source.endsWith('.log') can return no results even when events contain a source field ending in .log. If you must use it, match both the file extension and the event values (its example: source.endsWith('log') || source.endsWith('gz')), or collect without a source filter and filter later in Routes or Pipelines.

To verify that the Collector actually collects data, the page recommends a single run in Preview mode: select the Collector's Run action, select Preview, accept the other defaults, confirm with Run, and check the preview screen for data from Cribl Lake.

What the docs flag

• Sending events with only _raw to a Parquet Lake Dataset produces larger files than JSON and degrades search performance.
• Storing JSON objects in _raw is not an efficient way to get structured fields, even though Cribl Search can extract fields from valid JSON found in _raw.
• You can't use QuickConnect to configure Cribl Lake Collector Sources.
• Hybrid Stream Worker Groups that feed the collector must be running version 4.8 or higher.
• Filtering on source restricts which Lake files are pulled, so source.endsWith('.log') can return no results.

Frequently asked questions

What file formats does Cribl Lake store data in?

Cribl Lake stores data as either gzip-compressed JSON or Parquet files, and the format is chosen when you create a Lake Dataset. JSON with gzip compression produces smaller files when you don't need to query the data frequently; Parquet's columnar format enables faster queries for data you search and analyze regularly.

What should you remove from events for a storage-efficient JSON Lake Dataset?

A Pipeline in Cribl Stream can remove all top-level fields except _raw, which preserves the original event for compliance purposes. Keeping other top-level fields can improve search performance later, but results in larger file sizes in Cribl Lake.

Why must events sent to a Parquet Lake Dataset be structured?

Parquet stores data in columns and compresses repeated values efficiently, so events must arrive at Lake as parsed, structured data with multiple top-level fields for those benefits. Sending events with only _raw to a Parquet Lake Dataset produces larger files than JSON and degrades search performance.

Which Worker Groups can a Cribl Lake Collector pull from, and can QuickConnect configure it?

The Cribl Lake Collector can gather data from both Cribl-managed Cloud and customer-managed hybrid Stream Worker Groups, and hybrid Worker Groups must be running version 4.8 or higher. QuickConnect can't be used to configure Cribl Lake Collector Sources.