CrowdStrike FDR Data with Cribl Stream

What the source collects

The CrowdStrike FDR Source page documents a Cribl Stream source for the CrowdStrike Falcon Data Replicator (FDR) platform, whose data can then be sent to SIEM, threat-hunting, and other security tools and platforms. The page's header block names the source Type: Pull, TLS Support: Yes, Event Breaker Support: Yes.

The path is indirect: the FDR platform pulls data from Amazon S3 buckets maintained by CrowdStrike, so part of the configuration involves S3. At runtime, Worker Processes poll for messages from SQS. For importable configuration examples, the page links the CrowdStrike FDR Pack and the Exabeam Pack for CrowdStrike FDR.

Setting up the source

Setup follows the CrowdStrike FDR Source page: top bar, select Products, then Cribl Stream; under Worker Groups, select a Worker Group. Two paths: QuickConnect (Routing > QuickConnect > Add Source, Select Existing or Add New) or Routes (Data > Sources > Add Source).

In the New Source modal, under General Settings:

• Input ID: unique ID for this Source; the page's example is Endpoint42Investigation.
• Description: optional.
• Queue: the name, URL, or ARN of the SQS queue to read notifications from; non-AWS URLs must be formatted '{url}/myQueueName'. The value must be a JavaScript expression (which can evaluate to a constant), quoted or backticked, evaluated only at init time, such as when referencing a Global Variable.

Three optional settings follow:

• Filename filter: regex matching file names to download and process; defaults to .*.
• Region: the AWS Region where the S3 bucket and SQS queue are located; required unless Queue is a URL or ARN that includes a Region.
• Tags: filter and group Sources in the Stream UI, not added to processed events; names separated by a tab or hard return.

Optional Authentication, Assume Role, Processing, and Advanced sections follow, plus Connected Destinations; finish with Save, then Commit & Deploy.

Authentication. Choose a method: Auto (the default) uses the AWS SDK for JavaScript to obtain credentials in this order: Environment Variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY), IAM Identity Center (SSO), the shared credentials file (~/.aws/credentials), IAM roles for EC2 instances or ECS containers, a JSON file on disk, or other AWS SDK credential-provider classes. Auto works on AWS and elsewhere its credential sources exist; SSO providers like SAML and Okta can issue temporary credentials set in those environment variables. Manual, for Workers not running on AWS (for example a private cloud), takes static user-associated IAM credentials, directly or by reference; Access key falls back to env.AWS_ACCESS_KEY_ID, then the metadata endpoint, and Secret key to env.AWS_SECRET_ACCESS_KEY, then the metadata endpoint. Secret selects a stored Secret key pair from the internal secrets manager or, if enabled, an external KMS; a Create link configures one.

Assume Role. To reach resources in a different Region than Cribl Stream, set CRIBL_AWS_STS_REGION on the Worker Node to target that Region's AWS Security Token Service (STS) endpoint; CRIBL_AWS_STS_ENDPOINT specifies a custom endpoint and takes precedence if both are set, and an invalid Region or endpoint falls back to the global STS endpoint. Fields: Enable for Amazon S3 (defaults on), Enable for Amazon SQS (defaults off), AWS account ID (queue owner's account; empty if in the same account), AssumeRole ARN, and External ID.

Processing Settings. Custom Command (default off) passes this input's data to an external command before it continues downstream: the command consumes data via stdin, output is processed via stdout, and arguments are added one at a time, then dragged to resequence. Event Breakers applies rulesets, in order (default System Default Rule), to convert data streams to discrete events; the buffer timeout, in milliseconds, is how long the Event Breaker waits for new data on a channel before flushing, as-is, to the Routes: minimum 10 ms, default 10000 (10 sec), maximum 43200000 (12 hours). Fields defines or modifies fields with JavaScript expressions, similar to the Eval function, iterating in an advanced-mode modal against sample data. Pre-Processing optionally selects an existing Pipeline to run before the Routes; by default, events go to normal routing and event processing.

Advanced Settings. Enable checkpointing defaults off (Stream always starts a file from the beginning); on, it resumes a file where an error or restart interrupted it, adding Retries (ignored if Skip file on error is enabled). Other Settings, with the page's defaults: Endpoint (S3 service endpoint; if empty, built from the AWS Region; path-style URL, Stream appends the bucket name), Signature version (signs S3 requests; default v4, v2 available), Message limit (max messages an SQS poll returns; 1 to 10, default 1), Visibility timeout seconds (at-least hiding for messages retrieved by ReceiveMessage; default 21600 (6 hours), maximum 43200 (12 hours), 0 for none), Number of receivers (receiver processes; more means throughput at a CPU cost; default 1), Poll timeout (secs) (wait before polling again; minimum 1, default 10, maximum 20; UI warns under 5 seconds), Socket timeout (inactivity, seconds; default 300; raise for backpressure timeouts), Skip file on error (off, enabling retries; on, skips erroring files, e.g., corrupted ones), Reuse connections (on; can improve performance), Reject unauthorized certificates (on, the restrictive option), Encoding (unset means UTF-8, which might misinterpret multi-byte characters; ignored for Parquet files; UTF-16LE and Latin-1 also supported), and Environment (GitOps branch to enable the config on; empty means everywhere).

Connected Destinations: Send to Routes enables conditional routing, filtering, and cloning via the Routing table; QuickConnect sends data to one or more Destinations via independent, direct connections.

Once deployed, Worker Processes poll SQS for messages using the AWS SDK S3 APIs. Each polling call returns, by default, one message if any are available, changeable using Max messages; a call with no message times out after one second, and polling repeats. All Worker Processes participate, so files are collected and processed evenly across Workers. The modal's Live Data tab (Start Capture) and Logs tab, plus the Monitoring page, cover troubleshooting.

What the docs flag

The first flag is deployment-specific, per the source page: Azure-based Cribl.Cloud Worker Groups do not have AWS credentials, so Auto finds none and authentication fails, and the page says to select another method and supply your own AWS credentials. Manual and Secret cover exactly that non-AWS case.

SQS polling carries two caveats: Amazon SQS never returns more messages than the Message limit (though fewer might be returned), so the limit is an upper bound; and poll timeout cuts both ways, with short durations increasing the number, and thus the cost, of AWS requests (the UI warns under 5 seconds), and long durations slowing reactions to configuration changes and system restarts.

The 21600 second (6 hour) visibility timeout default is intentional: it gives each Worker Process time to decompress, stitch, and read large multi-part files without duplicate processing, and Cribl Stream extends it automatically until the initial request's files have been processed, notably for large files.

Checkpointing is the page's densest caveat area. It is off by default; checkpoint information lives in memory, is written to disk once per second and on Worker shutdown, and is kept at the Worker Process level, not shared. Destroyed Worker Node state (a Worker ceases to exist, which can happen during Cribl.Cloud upgrades) loses the information, and because checkpoints are not written on every event, an abrupt Worker Process stop loses unwritten checkpoints: resuming starts just after the last checkpoint written to disk, producing duplicate events. Keep checkpointing in mind for disaster recovery planning, the page adds.

Two smaller notes: unset encoding defaults to UTF-8, which might misinterpret multi-byte characters, and is ignored for Parquet files; and Retries is ignored when Skip file on error is toggled on.

Frequently asked questions

How does Cribl Stream receive CrowdStrike FDR data?

It polls for messages from SQS about files in the Amazon S3 buckets maintained by CrowdStrike, using the AWS SDK S3 APIs. The page lists the Source as Pull type, with TLS support and Event Breaker support, and each polling call returns one message by default when any are available.

Which authentication methods can the CrowdStrike FDR Source in Cribl Stream use?

The Source offers three methods: Auto, Manual, and Secret. Auto is the default, using the AWS SDK for JavaScript to obtain credentials in a documented order beginning with the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. Manual accepts static IAM credentials for Workers not running on AWS; Secret selects a stored secret key pair from the internal secrets manager or an external KMS.

What does checkpointing do in the CrowdStrike FDR Source, and when does the page say it produces duplicate events?

It resumes a file from the point where processing was interrupted, after an error or restart. It is off by default; checkpoints live in memory, are written to disk once per second and on Worker shutdown, at the Worker Process level. If a Worker ceases to exist or a Worker Process stops abruptly, checkpoints are lost, and resuming from the last checkpoint written to disk produces duplicate events.

What defaults do the SQS polling settings carry on the CrowdStrike FDR Source in Cribl Stream?

Message limit defaults to 1 (acceptable values 1 to 10), Visibility timeout seconds to 21600 (6 hours, maximum 43200 or 12 hours), and Poll timeout to 10 seconds (minimum 1, maximum 20). Amazon SQS never returns more messages than the limit, though fewer might be returned, and the visibility timeout is extended until the initial request files are processed.