Amazon Security Lake with Cribl Stream

What Amazon Security Lake is

Amazon Security Lake is AWS's security data lake service. Security data lands in one shared lake, backed by S3 buckets and stored as Parquet files, instead of each product pushing its own format into analytics tools.

That structure is OCSF, the Open Cybersecurity Schema Framework: an open schema with classes for sign-ins, detections, process activity, and identity events, each with standardized fields, making events from different products comparable. AWS Security Hub is a different service, a posture dashboard rather than a lake.

Sending data to it

The Amazon Security Lake Destination in Cribl Stream delivers data to Amazon Security Lake, which the destination reference describes as fully managed. It is a Non-Streaming type with TLS support, lists PQ support as No, and is unavailable on Windows and macOS.

Security Lake ingests events described by OCSF and conveyed as Parquet files, and this Destination sends those Parquet files in batches. Cribl Stream does not read or replay the data it sends. The page documents how files are written rather than a normalization pass, and pins no OCSF version to Stream's output.

New Destination general settings:

• Output ID, a unique name (clones get -CLONE appended), and an optional Description.
• S3 bucket name: the destination bucket created when setting up Security Lake; a constant or JavaScript expression evaluated only at init time (event-level variables unavailable).
• Region: where Security Lake is located. Account ID: the AWS account whose admin enabled it. Custom source name: the custom source created in Security Lake, an S3 bucket for a single OCSF event class.
• Staging location: where files buffer locally before compression and delivery (not shown on Cribl.Cloud-managed Worker Nodes).

The Authentication Method must be left at its default, Auto: short-lived credentials from the EC2-attached IAM role via the AWS instance's metadata service, or from the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, working only when running on AWS. Assume Role takes values from the custom source: AssumeRole ARN (the Provider role ARN), External ID (the trust relationship's ExternalId), and Duration (seconds), default 3600, minimum 900, maximum 43200.

Processing Settings add a Pipeline drop-down for data processed before sending, plus a System fields list (none by default) such as cribl_host and cribl_output.

Parquet Settings: the Automatic schema toggle (off by default) generates the schema from each file's events; when off, the Parquet schema drop-down exposes the schema library. Parquet version defaults to 2.6. Optional Metadata key-value pairs go into file metadata; the page's example tags events as the security findings OCSF category with Key OCSF Event Class and Value 2001.

Advanced Settings:

• File close conditions, any one of which closes and moves a file: File size limit (MB) (default 32), File open time limit (sec) (default 300), Idle time limit (sec) (default 30), Open file limit (default 100).
• Concurrent file parts limit (default 4, highest 10): 1 sends the whole file at once; 2 or higher needs the multipart IAM permissions.
• Add Output ID and Remove empty staging directories are on by default; Enable dead-lettering caps retries at Maximum retry limit (default 20), moving undeliverable files to the Dead-letter location (default $CRIBL_HOME/state/outputs/dead-letter).
• Endpoint (empty defaults to AWS' Region-specific endpoint), Storage class (default Standard), Server-side encryption (no encryption by default; Amazon S3 Managed Key or AWS KMS Managed Key), Signature version (default v4).

The Destination's always-needed IAM permissions, copied from the destination reference one per line:

s3:ListBucket
s3:GetBucketLocation
s3:PutObject

Multipart uploads need two more, copied from the destination reference:

kms:GenerateDataKey
kms:Decrypt

Reading from it

Cribl Stream supports receiving data from Amazon Security Lake. The source reference lists a Pull type with TLS support (secure API) and Event Breaker support; Parquet reading requires Linux, via the .parquet, .parq, or .pqt filename extension.

The page describes Security Lake as aggregating, normalizing, and storing security-related data from an AWS system, outputting it in OCSF. Use it to transform, trim, or shape incoming data for a downstream Destination, or to collect from multiple AWS environments or Regions.

The Source functions nearly identically to the Amazon S3 Source. Both receive data via Security Lake event subscriptions or S3 bucket event notifications through SQS; the underlying data formats differ, and Stream supports both. Pulling data from HTTPS endpoints is not currently supported. Security Lake pushes the Parquet object to a queue in an S3 bucket and notifies Cribl Stream of its location; Stream pulls and deserializes it.

General Settings: the Input ID (clones get -CLONE appended) and the Queue, the name, URL, or ARN of the SQS queue. A non-AWS URL uses this format, copied from the source reference:

{url}/<queueName>
https://host:port/<queueName>

Optional Settings: Filename filter, a regex defaulting to .* evaluated against the S3 key's full path; Region, required unless the Queue entry includes one; and Tags for filtering and grouping in the UI.

Authentication offers three methods. Auto, the default, has the AWS SDK for JavaScript attempt credentials in a fixed order (environment variables, IAM Identity Center (SSO), the shared credentials file, IAM Roles for Amazon EC2/ECS, a JSON file on disk, and other providers). Manual, when not running on AWS, takes a static Access key and Secret key. Secret, also when not running on AWS, selects a stored Secret key pair from the internal secrets manager or an external KMS, with its C.Secret() reference expressions copied from the source reference:

${C.Secret('myKeyPair', 'keypair').apiKey}
${C.Secret('myKeyPair', 'keypair').secretKey}

Assume Role reaches resources in a different Region: Enable for Amazon S3 on by default, Enable for Amazon SQS off by default, Duration (seconds) defaulting to 3600 (the Destination's 900 to 43200 bounds), and AWS account ID naming the queue owner in a different account.

Processing: Event Breakers apply rulesets in order (default System Default Rule, buffer timeout defaulting to 10000 ms or 10 sec); Fields adds or modifies fields with JavaScript expressions; Include notification metadata adds the S3 event notification payload to a __sqsMetadata field on every event from the corresponding S3 file.

Advanced Settings:

• Enable checkpointing, off by default; Retries counts file retries after an error or restart, ignored if Skip file on error is on.
• Endpoint (empty defaults to the AWS Region-specific endpoint; Stream adds the bucket name to the URL path); Poll timeout (secs) (default 10, minimum 1, maximum 20).
• Parquet chunk size limit (MB) (default 5, range 1 to 100); Parquet chunk download timeout (seconds) (default 600, range 1 second to 3600, processing ends if a chunk misses the window).

Checkpointing, off by default, tracks how many events were processed per file; on resume the Source skips the first N seen and starts at N+1. Checkpoints live in memory, are written to disk once per second and on Worker shutdown, and are never shared between Worker Processes or Nodes.

The page offers Job Inspector, Live Data, Logs, and Monitoring for troubleshooting. Its internal fields are __final, __inputId, __source, and _time.

What the docs flag

• Destination auth is AWS-only. The Destination's Auto method works only when running on AWS, and the destination page documents no alternative method.
• Schema edits do not propagate. Modifying an existing schema in the Schema Library does not propagate your modifications to the Destination's copy of the schema. Cribl strongly recommends cloning first and giving the clone a distinct name; otherwise you still must re-select it in the Parquet schema drop-down and click Save.
• Cribl.Cloud differs. The Staging location field is not shown on Cribl.Cloud-managed Worker Nodes, and Cribl.Cloud setup differs from the documented procedures, per Cribl's #packs channel guidance. On the Source side, Azure-based Cribl.Cloud Worker Groups have no AWS credentials, so Auto fails; select another method and supply your own.
• Do not retune the default input transformer. Changing how Security Lake represents data could cause issues with Cribl Stream consuming it, so only advanced AWS administrators should change it.
• Checkpointing is not bulletproof. When Worker Node state is destroyed, checkpoints are lost (Cribl.Cloud upgrades can do this), and because they are not written on every event, an abrupt stop resumes from the last disk checkpoint, producing duplicate events.

Frequently asked questions

How does Cribl Stream send data to Amazon Security Lake?

Through the Amazon Security Lake Destination, which sends Parquet files in batches to the S3 bucket you name in its S3 bucket name setting. It is Non-Streaming, and its page documents file mechanics, not an OCSF normalization pass.

Which authentication method must be left selected on the Cribl Stream Amazon Security Lake Destination?

The default Auto method, which is the only method the Destination page documents. It uses the AWS instance metadata service to obtain short-lived credentials from the EC2 IAM role, or the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, and works only when running on AWS.

Can the Cribl Stream Amazon Security Lake Source read Parquet files on Windows or macOS?

No, Cribl Stream must run on Linux for this Source to read Parquet files, which it identifies by the .parquet, .parq, or .pqt filename extension. The Destination carries its own platform note: it is unavailable on Windows and macOS.

Why do the Cribl docs recommend cloning a Parquet schema before modifying it for the Amazon Security Lake Destination?

Modifying an existing schema in the Schema Library does not propagate your modifications to the Destination's copy of the schema. Cloning and renaming removes any doubt that the Destination will use the new version.