Elastic Cloud with Cribl Stream

What it is for

Cribl Stream can send events to Elastic Cloud through a dedicated Elastic Cloud Destination, listed by the docs as Type Streaming with TLS Support: Yes and PQ Support: Yes.

The page reserves the Elasticsearch Destination for when you need flexibility to support both self-hosted and Elastic Cloud deployments and require advanced configuration options.

Configuring the destination

You configure it from a Worker Group (Products, Cribl Stream) through QuickConnect or Routes; both paths end with Add Destination, and the QuickConnect path additionally offers Select Existing or Add New.

General Settings take an Output ID (a unique name identifying this Destination definition), an optional Description, the Cloud ID of your Elastic Cloud environment (found on its Deployments overview page), and a Data stream or index set to a JavaScript expression evaluated for each event; it can evaluate to a constant value, must be enclosed in quotes or backticks, and an event's __index field can overwrite it.

Authentication offers four methods, per the Elastic Cloud Destination documentation:

• Manual: credentials entered directly in the resulting Username and Password fields.
• Secret: a Credentials secret drop-down to select a stored secret, with a Create link to store a new, reusable secret.
• Manual API Key: an API key field for your Elasticsearch API key.
• Secret API Key: an API key (text secret) drop-down, with a Create link to store a new, reusable secret.

Optional Settings include Backpressure behavior (whether to block, drop, or queue events when all receivers are exerting backpressure; default Block) and Tags (filter and group Destinations on the Destinations page, not added to processed events, separated by a tab or hard return). Then optionally adjust Persistent Queue, Processing, Retries, and Advanced settings, select Save, then Commit & Deploy.

The Persistent Queue Settings tab displays when Backpressure behavior in General settings is set to Persistent Queue, buffering and preserving incoming events when a downstream Destination has an outage or experiences backpressure. Some settings might not appear if the Worker Group PQ storage mode is Network filesystem or AWS S3, where that behavior is managed at the Worker Group level. On Cribl-managed Cloud Workers with an Enterprise plan, the tab exposes only Mode, Backpressure duration limit, and the destructive Clear Persistent Queue button. Queue settings:

• Mode: when the queue engages. Error stores data on disk when the Destination is unavailable or in error; Backpressure while backpressure is detected; Always On queues and stores all data on disk immediately, even without backpressure. An invalid Mode automatically switches to Error, for example on a version that does not support other modes (older than 4.9.0) or on a nonexistent YAML value.
• Queue size limit: maximum disk space per Worker Process; defaults to 5 GB, up to 1 TB unless a different Worker Process PQ size limit is configured; at the limit, queueing stops and the Queue-full behavior applies.
• Queue-full behavior: when the disk is low or at full capacity, Block refuses new data until the receiver is ready; Drop new data discards new events until backpressure resolves.
• Buffer size limit (bytes): maximum memory buffered before flushing to disk; defaults to 1MB; valid range 64KB to 10MB, with out-of-range values rounded to the nearest limit.
• Backpressure duration limit: displays when Mode is Backpressure; seconds of backpressure before queueing to persistent storage; defaults to 30.
• Strict ordering: on by default, giving FIFO forwarding with the queue flushing every 10 seconds; off prioritizes new events and shows Drain rate limit (EPS), whose default of 0 disables throttling.
• Clear Persistent Queue: Cloud Enterprise only; permanently deletes queued files without delivering them; appears only after the Output ID is defined.

Processing Settings (Post-Processing) set the Pipeline or Pack that processes data before it is sent, and System fields added automatically to events using this output; the default includes cribl_pipe, wildcards are supported, and other available fields include cribl_host, cribl_input, and cribl_output.

Honor Retry-After header, when on, honors the header provided the stated delay is no longer than 180 seconds and takes precedence over all other Retries options; off ignores all such headers. Default Retry Behavior applies an exponential backoff retry for certain failed requests, overridable per error code; the default rules are copied from the Elastic Cloud Destination page:

| Status Code | Action |
| --- | --- |
| Any in the `1xx`, `3xx` series| Drop the request |
| `401`, `403`, `408`, `429` errors| Retry in memory or persistent queue if enabled, and initiate backpressure after limit is reached |
| Any other `4xx` series| Drop the request |
| Any in the `5xx` series| Retry in memory or persistent queue if enabled, and initiate backpressure after limit is reached |

A 429 (Too Many Requests) not listed in the retry settings is treated as a non-retryable 4xx error and dropped; include 429 in Settings for failed HTTP requests to retry it. Each listed code sets a Pre-backoff interval (ms, default 1000), a Backoff multiplier (default 2), and a Backoff limit (ms, default 10,000, maximum 180,000); Retry timed-out HTTP requests exposes the same three settings.

Advanced Settings: Request timeout defaults to 30 seconds; Request concurrency to 5 (minimum 1, maximum 32) concurrent requests per Worker Process; Body size limit (KB) to 4096 KB (the actual body might exceed it, since the Destination adds bytes); Events-per-request limit to 0, unlimited; Flush period (s) to 1. Extra HTTP headers and Extra parameters pass name-value pairs (headers sent encrypted), with a pipeline parameter available for Elastic ingest pipelines. Elastic pipeline optionally names an ingest pipeline, overridable by expression from the Elasticsearch Source. Include document_id off omits the field for Elastic TSDS (time series data stream) events. Failed request logging mode: None (default), Payload, or Payload + Headers, with Safe Headers listing the headers to log (empty redacts all). Environment (GitOps) scopes the configuration to one Git branch, or enables it everywhere when empty.

Using it

Recommended settings for Elastic Cloud Serverless deployments, tuned to the serverless autoscaler and aligned with the Elastic Agent Scale preset, are copied from the Elastic Cloud Destination page:

| Setting | Value |
| ------- | ----- |
| **Request timeout** | `60` seconds |
| **Request concurrency** | `4` |
| **Body size limit (KB)** | `5120` KB |
| **Events-per-request limit** | `3000` |
| **Flush period (s)** | `60` seconds |

Keep Request concurrency low to avoid outpacing the autoscaler before it has warm shards; the shorter Request timeout surfaces HTTP 429 responses quickly; the 60-second Flush period balances batching efficiency against data freshness. The Destination normalizes _time to @timestamp at millisecond resolution and sets host.name to host.

What the docs flag

• Always use the Elastic Cloud Destination for Serverless deployments; the Elasticsearch Destination does not work with Serverless due to TLS and authentication incompatibilities.
• HTTP 429 responses during autoscaler ramp-up are expected; enable retries with exponential backoff and do not treat 429 as a fatal error. During cold start (a new data stream, the first hour of ingestion), consider dropping Request concurrency to 2 and Body size limit (KB) to 2048 KB until the autoscaler has warm shards, then restore the values above.
• Destinations created before the 4.16.0 release do not support the default 4xx series retry behavior; reconfigure the Destination to support it.
• The Buffer size limit (bytes) setting replaced the deprecated Buffer size limit (events) setting as of version 4.18.0, defaulting to 1MB for upgraded Worker Groups and Fleets.
• On Cribl-managed Cloud Workers with an Enterprise plan, a 1 GB maximum queue size is automatically allocated per PQ-enabled Destination, per Worker Process, on outbound uncompressed data. The limit is not configurable; a full queue blocks outbound data, and only a hybrid Group configures queue size, compression, and queue-full fallback behavior.
• Use the Clear Persistent Queue button with caution to avoid data loss.

Frequently asked questions

What authentication methods does the Elastic Cloud Destination in Cribl Stream support?

Four: Manual (username and password entered directly), Secret (a stored secret), Manual API Key (an Elasticsearch API key entered directly), and Secret API Key (a stored text secret). The Secret and Secret API Key methods each offer a Create link to store a new, reusable secret.

When does the Persistent Queue Settings tab appear on the Elastic Cloud Destination?

It displays when the Backpressure behavior option in General settings is set to Persistent Queue. The queue there buffers and preserves incoming events when a downstream Destination has an outage or experiences backpressure.

Which HTTP status codes does the Elastic Cloud Destination retry by default?

It retries 401, 403, 408, and 429 errors and anything in the 5xx series, in memory or the persistent queue if enabled, and it drops the 1xx and 3xx series plus any other 4xx. A 429 not listed in the failed HTTP request retry settings is treated as a non-retryable 4xx error and dropped.

What settings does the documentation recommend for Elastic Cloud Serverless?

A Request timeout of 60 seconds, Request concurrency of 4, a Body size limit of 5120 KB, an Events-per-request limit of 3000, and a Flush period of 60 seconds, aligned with the Elastic Agent Scale preset. During cold start, consider dropping Request concurrency to 2 and Body size limit to 2048 KB until the autoscaler has warm shards.