Cribl Stream Persistent Queues: Sources, Destinations, and Shared Storage

What persistent queues are for

Cribl Stream's persistent queues (PQ) hold data when the pipe in front of a Destination clogs. The About Persistent Queues page describes them as supplementing in-memory queues to minimize data loss: when a backpressure event occurs, Cribl Stream writes data to persistent queue storage, then forwards the queued data once the outage resolves.

The same page targets three situations: a downstream Destination receiver that is unreachable, a slow-to-respond downstream receiver (Cribl Stream 4.9 and newer only), and a Source sending more data than Cribl Stream can process. It fits deployments with outages longer than short-term memory queues can sustain (typically a few seconds), organizations with a low tolerance for data loss, and upstream senders with no buffering of their own, such as UDP senders.

Coverage is specific: HTTP-based, TCP load-balanced, and single-receiver Destinations, plus Push Sources, support persistent queues, while Pull and Collector Sources do not. The page also draws the distinction people conflate: persistent queues are specifically on-disk storage, while backpressure queues can be in-memory or on-disk buffers. Use backpressure queueing on either Cribl Stream or the sender or receiver, not both, since double queueing can cause data latency.

Source-side queue settings

The Optimize Source Persistent Queues guide lists the triggers: error conditions, degraded performance in downstream Cribl resources, and backpressure triggered by external receivers; when the queue engages, Cribl Stream writes the data to local or shared storage until the event resolves.

The Mode setting decides when the queue writes. Always On, the default since Cribl Stream 4.1, disk-buffers all events, flushing every 10 seconds while the queue drains. Smart engages only on backpressure from downstream receivers or internal processes, queueing events before any pre-processing Pipeline takes effect. Smart mode is deprecated for hybrid and on-prem deployments, with automatic migration to Always On in January 2027.

With local file system storage (the default), the Source's Persistent Queue Settings tab exposes these options and docs defaults:

• Buffer size limit (bytes): maximum memory buffered before flushing to disk; 64KB to 10MB, default 1 MB; shared across all connections, not per Worker Process.
• Commit frequency: events sent downstream before Cribl Stream commits that it has read them; default 42.
• File size limit: maximum data per queue file before it closes; default 10 MB, at least the Buffer size limit.
• Queue size limit: maximum disk space per Worker Process; default 5 GB, up to 1 TB; raising the Worker Process PQ size limit (maxPQSize in limits.yml) needs Cribl Support guidance.
• Queue file path: default $CRIBL_HOME/state/queues, with /<worker-id>/inputs/<input-id> appended.
• Compression: None (default) or Gzip, applied after the queue file closes.
• Queue-full behavior: Block (default) holds incoming data until the queue has capacity; Drop new data discards incoming events, reported in the pq.dropped_events metric, while leaving queued contents unchanged.

The guide strongly recommends persistent queues for UDP Sources: UDP senders cannot be told to slow down, so if Cribl Stream cannot keep up it drops new events. Enabling Source persistent queue on a UDP Source bypasses the Advanced Settings max buffer size (events) field, hidden and ignored in Cribl Stream 4.10 and newer for UDP Sources.

Destination-side queue settings

The Optimize Destination Persistent Queues guide lays out the order of operations: under normal conditions, Destinations store data in in-memory buffers, and when those buffers fill, a backpressure event triggers the Destination's Backpressure behavior: Block holds all data and returns block signals upstream, Drop Events discards new events, and Persistent Queue engages the on-disk queue if the Destination supports it.

Destinations offer three modes, one more than the Source side: Error queues and stores data when the Destination is unavailable or in an error state; Backpressure queues while Cribl Stream detects backpressure until the event resolves; Always On immediately queues all data to persistent storage. An invalid Mode value falls back to Error, as do Nodes older than 4.9.0, and a Backpressure duration limit (default 30 seconds) gates when Backpressure-mode queuing starts.

When the event ends, the queue drains in FIFO order, flushing every 10 seconds, earliest events first. In Cribl Stream 4.1 and newer you can disable Strict Ordering to prioritize new events instead; that requires a Drain rate limit (EPS), which the docs recommend setting near 10% of a single Worker Process's throughput.

Two engagement specifics: with load-balanced Destinations, the queue engages only when all receivers are slow or blocking, since one healthy receiver prevents engagement; with Kafka-based Destinations, it engages when Cribl Stream cannot reach the brokers. Delivery is at least once, not exactly once, so duplicates can emerge.

Sizing gets concrete here: each PQ-enabled Source and Destination keeps its own queue on every Worker Process, and each Destination queues its own copy, so fanning out multiplies disk. The guide's worked example sizes a Worker Group receiving 240 GB per day across 8 Worker Processes, fanned out to 2 PQ-enabled Destinations, for a 4-hour outage:

240 GB/day = 10 GB/hour
4-hour outage: 10 GB/hour x 4 hours = 40 GB
Per Worker Process: 40 GB / 8 = 5 GB
Queue size limit: at least 5 GB for each Destination
Disk per Worker Node: 8 x 2 x 5 GB = 80 GB

Set the Queue size limit at or above the per-Destination figure the math produces; if it is lower than the volume an outage produces, queues reach the limit and the Queue-full behavior applies before the event ends.

Shared storage mode

By default, persistent queues write to each Worker Node's local disk; if that Node goes down before the queue drains, the data stays there until the same Node returns. The Persistent Queue Shared Storage page describes the alternative: a shared backing store, either a Network filesystem (NFS) volume or an AWS S3 bucket, readable by all Worker Nodes in the Group. When a Node disappears, survivors recover and drain its queues.

Use shared storage when Worker Nodes are replaced, scaled down, or fail during a backpressure event, when queued data must survive the loss of a specific Node, or in elastic environments such as Kubernetes autoscaling. Use local storage when Node identities are stable. Availability is narrow: shared PQ storage and orphan management work only on customer-managed and hybrid Worker Groups and require an Enterprise license, Cribl Edge uses local storage only, and on Cribl.Cloud-provisioned Groups Cribl manages shared storage itself.

Orphan management controls revival. An orphan is queued data on shared storage from a Worker Node no longer active in the Group; a restart that preserves Node identity is not an orphan. When a Node has been absent longer than the Orphan detection threshold, Cribl Stream can reassign its orphaned queues to surviving Nodes, which drain them. The settings live at Worker Group Settings > System: Scan interval (minutes), default 1; Orphan detection threshold (minutes), default 20; plus a disable toggle for planned maintenance.

Two migration gotchas: enabling shared storage automatically copies existing local PQ data to the shared store, so plan for that window; switching back to local does not migrate data, so drain the shared queues or use Clear persistent queue first.

What the docs warn about

The warnings on these pages include:

• Orphaned data: turning off persistent queue, or shutting a Source or Destination down, before the queues drain orphans any events still in the store. On local storage the data is inaccessible unless you re-enable the queue on the same Worker Node; on shared storage it stays in the S3 bucket or NFS mount.
• Destructive clear: the Clear Persistent Queue button frees disk space by deleting queued data without sending it on to receivers.
• Disk floor: when free space falls below the Min free disk space threshold (default 5 GB), Cribl Stream stops maintaining persistent queues and data loss begins.
• No double queueing: enabling persistent queue where the sender or receiver already buffers events on disk is redundant, and running both can cause data latency.
• Fixed size in the cloud: on Cribl.Cloud, Cribl allocates up to 1 GB of disk space per Source or Destination per Worker Process, uncompressed, and blocks data flow if the queue fills. You cannot change the limit; hybrid and on-prem deployments are the path to configuring size, compression, and modes.

Frequently asked questions

What is a Cribl Stream persistent queue?

A Cribl Stream persistent queue is an on-disk buffer that supplements in-memory queues to minimize data loss during a backpressure event. Cribl Stream writes data to persistent queue storage during the outage, then forwards it once the outage resolves.

What modes can a Cribl Stream Destination persistent queue use?

A Cribl Stream Destination persistent queue can run in Error, Backpressure, or Always On mode: Error queues data when the Destination is unavailable or in an error state, Backpressure queues while Cribl Stream detects backpressure, and Always On queues all data to persistent storage immediately.

What happens when a Cribl Stream persistent queue fills up?

Cribl Stream applies the configured Queue-full behavior, which is either Block or Drop new data. Block holds incoming data until the queue has capacity and returns block signals to senders that support backpressure signaling, while Drop new data discards incoming events and leaves the queued contents unchanged.

How does shared storage change Cribl Stream persistent queues?

With shared storage, all Worker Nodes in a Worker Group store queue data on a common NFS volume or AWS S3 bucket. Surviving Nodes can recover and drain the queues of a Node that disappears, and orphan management reassigns those queues once a Node is absent longer than the Orphan detection threshold (20 minutes default).