Pulling Data from Amazon S3 Buckets with Cribl Stream

The S3 Source

Cribl Stream supports receiving data from Amazon S3 buckets. The Amazon S3 Source receives data via either of two methods: S3 Bucket event notifications through Amazon Simple Queue Service (SQS), or Amazon Security Lake event subscriptions. Both send data objects to a queue the same way, but their underlying data formats differ, and Cribl Stream supports both. The page marks the Source as a Pull type with TLS support (secure API); Parquet reading requires Linux (only), by the .parquet, .parq, or .pqt filename extension.

The source S3 bucket must be configured to send s3:ObjectCreated:* events to an SQS queue, either directly or via SNS. The page's walkthrough: create a Standard SQS Queue, replace its access policy, and add a notification configuration publishing s3:ObjectCreated:* to the queue. Its two example policies differ only at line 9, public access versus S3-only access; the restrictive example, copied from the source reference:

{
 "Version": "2012-10-17",
 "Id": "example-ID",
 "Statement": [
  {
   "Sid": "<SID name>",
   "Effect": "Allow",
   "Principal": {
    "Service":"s3.amazonaws.com"  
   },
   "Action": [
    "SQS:SendMessage"
   ],
   "Resource": "example-SQS-queue-ARN",
   "Condition": {
      "ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:example-bucket-name" }
   }
  }
 ]
}

In the New Source modal, General Settings: an Input ID (clones get -CLONE appended), an optional Description, and the Queue: the name, URL, or ARN of the SQS queue to read events from. A non-AWS URL must use the format {url}/<queueName>, and the value must be a JavaScript expression quoted in single, double, or backticks. Optional Settings add a Filename filter (a regex over the S3 key's full path that defaulting to .*), Region (required unless the Queue is a URL or ARN that includes a Region), and Tags (UI-only, not added to processed events).

The Authentication method drop-down offers three options. Auto (default) uses the AWS SDK for JavaScript to obtain credentials in this order: Environment Variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), IAM Identity Center (SSO), the Shared Credentials File (~/.aws/credentials), IAM Roles for Amazon EC2/ECS, a JSON File on Disk, and other credential-provider classes. Auto works on AWS and elsewhere when those credentials exist.

Manual, for Workers not in an AWS VPC (such as a private cloud), takes static user-associated IAM credentials directly or by reference: Access key (falls back to env.AWS_ACCESS_KEY_ID or the metadata endpoint for IAM role credentials) and Secret key (falls back to env.AWS_SECRET_ACCESS_KEY or the metadata endpoint for IAM credentials). The page's C.Secret() reference values, copied from the source reference:

${C.Secret('myKeyPair', 'keypair').apiKey}
${C.Secret('myKeyPair', 'keypair').secretKey}

Secret, also for non-AWS deployments, exposes a Secret key pair drop-down for a key pair configured in Cribl Stream's internal secrets manager or, if enabled, an external KMS.

The Source supports independent, multi-hop AssumeRole chains for S3 and SQS. Each chain is an ordered list of STS AssumeRole hops using the previous hop's resolved credentials (max 3 per chain; an empty chain disables assumption for that service). Each hop has an AssumeRole ARN, an External ID, a Duration (seconds) (minimum 900, maximum 43200, default 3600), and an optional Session name (2-64 characters, alphanumeric plus +, =, ,, ., @, -, defaulting to a generated value); the SQS chain adds an AWS account ID (empty when same-account).

For cross-region AssumeRole, CRIBL_AWS_STS_REGION sets the STS region and CRIBL_AWS_STS_ENDPOINT a custom endpoint (endpoint precedence when both are set; invalid values fall back to the global STS endpoint). Advanced Settings adds Endpoint (empty defaults to AWS's Region-specific endpoint; otherwise an S3-compatible endpoint, with the bucket name added to the URL path), Signature version (default v4, signs SQS requests), and Override credentials for S3: on, the S3 client uses its own base Authentication method; off (default), S3 inherits SQS's top-level credentials.

How it reads data

Workers poll messages from SQS: the call returns messages if they are available, or times out after 1 second if none are. Each Worker gets its share of the load; by default a single poll request returns at most 1 message. The Message limit setting (called Max messages in the page's pull and best-practices sections) accepts 1 to 10, defaulting to 1. Poll timeout (secs) waits before each re-poll: minimum 1, default 10, maximum 20; short durations mean more, costlier AWS requests; long durations slow reaction to configuration changes and restarts.

Visibility timeout seconds (default 600) hides retrieved messages from later retrieve requests; Cribl Stream extends it until the request's files are processed. Number of receivers (default 1) sets how many receiver processes run; the higher the number, the better the throughput, at the expense of CPU overhead.

Restart and error behavior centers on Enable checkpointing (Advanced Settings, off by default): it resumes a file at the interruption point, skipping the first N events seen and starting at event N+1, minimizing duplicate events downstream. Checkpoints live in memory, are written to disk once per second and on Worker shutdown, and are not shared between Worker Processes or Nodes. It exposes Retries, ignored when Skip file on error is on (off by default, which enables retries).

Cribl Stream ingests compressed S3 files that carry the x-gzip MIME type, end in .gz, .tgz, .tar.gz, or .tar, or can be uncompressed by zlib.gunzip. For Parquet (Linux only), Parquet chunk size limit (MB) defaults to 5 MB (1 to 100) and Parquet chunk download timeout (seconds) to 600 (1 to 3600 seconds); a required chunk that misses the timeout ends processing. Encoding, if not set, defaults to UTF-8 but might incorrectly interpret multi-byte characters; it is ignored for Parquet files, and UTF-16LE and Latin-1 are also supported.

What the docs flag

SQS messages are deleted after they are read, unless an error occurs, in which case Cribl Stream will retry. Files that fail the Filename Filter are ignored, but their SQS events are still read and then deleted from the queue, unavailable to other S3 Sources targeting the same queue. The page's alternatives: a dedicated SQS queue (preferred and recommended), or a broad filter on a single Source with pre-processing Pipelines and/or Route filters.

Cribl Stream does not support data preview, collection, or replay from S3 Glacier or S3 Deep Glacier storage classes, whose stated retrieval lags (variously minutes to 48 hours) cannot guarantee data availability when the Collector needs it. It does support them from S3 Glacier Instant Retrieval when using the S3 Intelligent-Tiering storage class.

Permissions the page requires: on the S3 bucket, s3:GetObject and s3:ListBucket (plus s3:GetObjectTagging and s3:PutObjectTagging when Tag after processing is on); on the SQS queue, sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility, sqs:GetQueueAttributes, and sqs:GetQueueUrl. Encrypted data, such as CloudTrail logs, additionally needs the KMS Key Policy to grant the IAM user or role kms:Decrypt.

Azure-based Cribl.Cloud Worker Groups do not have AWS credentials, so Auto finds none and authentication fails; select another authentication method and supply your own AWS credentials.

Checkpointing caveats: when Worker Node state is destroyed (Cribl.Cloud upgrades can do this), checkpoint information is lost, and because checkpoints are not written to disk on every event, an abruptly stopped Worker Process resumes from just after the last disk checkpoint, producing duplicate events.

Common issues: "InvalidClientTokenId" or "The security token included in the request is invalid" can occur with AssumeRole in a non-default region (for example, eu-central-2), since such regions do not accept STS tokens minted by the global endpoint; the page's fix is CRIBL_AWS_STS_REGION (for example, eu-central-1) or CRIBL_AWS_STS_ENDPOINT. A persistent "Inaccessible host: 'sqs.us-east-1.amazonaws.com'" error can be a TLS negotiation failure (the page notes AWS SQS currently does not support TLS 1.3).

The page's best practices: use IAM Roles whenever possible on AWS, use a Filename Filter even though optional, and raise Number of Receivers and/or Max messages for throughput (both default to 1; more throughput means more CPU).

Frequently asked questions

How does the Amazon S3 Source in Cribl Stream receive data from a bucket?

It does not read the bucket directly; the bucket must be configured to send s3:ObjectCreated:* events to an SQS queue, and Cribl Stream's Workers poll that queue for messages. Its Queue setting names the queue by name, URL, or ARN.

Which AWS credentials can the Cribl Stream S3 Source use?

Three authentication methods are documented: Auto, Manual, and Secret. Auto walks the AWS SDK for JavaScript chain (environment variables, IAM Identity Center (SSO), shared credentials file, IAM Roles for EC2/ECS, JSON file on disk, other provider classes); Manual takes a static access key and secret key; Secret selects a stored key pair. AssumeRole chains keep S3 and SQS credential chains independent, and Override credentials for S3 gives the S3 client its own base Authentication method.

What should be enabled in Cribl Stream so the S3 Source does not collect duplicate events after a worker restart?

Enable checkpointing under Advanced Settings: it resumes a file where an error or restart occurred, skipping the first N events seen and starting at event N+1. The toggle is off by default; checkpoints live in memory, written to disk once per second and on Worker shutdown.

What happens to S3 files that do not match the Filename Filter in Cribl Stream's S3 Source?

They are ignored by Cribl Stream, but their SQS events are still read and then deleted from the queue, so they are no longer available to other S3 Sources targeting the same SQS queue. The page recommends a dedicated SQS queue (preferred and recommended) or a broad single-Source filter with pre-processing Pipelines and/or Route filters.