Backpressure in Cribl Stream: What Slows Your Sources Down

What backpressure is

When a Destination experiences backpressure, Cribl Stream must manage the incoming data flow to prevent overwhelming the system. The Backpressure Impacts to Sources page in the docs describes what happens next when the Backpressure behavior is set to Block and persistent queue is not in use: backpressure propagates back through the data Pipeline to the Sources, and the impact on data flow differs depending on the Source type. The same page points to the Data Delivery to Unreachable Destinations section of the Destinations documentation for the available backpressure behavior options.

Two properties determine how a Source feels the pressure. First, the speed of propagation depends on buffer sizes throughout the Pipeline: a brief backpressure event might not immediately affect all Sources, but sustained backpressure will eventually impact all connected senders. Second, whether the Source type supports backpressure signaling at all. Supported Sources block their senders and data queues on the sender side. Unsupported Sources, such as UDP-based ones, drop data.

How it reaches your sources

The Destination does not directly notify individual Sources about backpressure. Instead, backpressure propagates indirectly: the Cribl Stream event processors stop reading, the Source buffers fill, and the Source signals upstream senders where supported. The page documents this as a chain reaction that occurs when a Destination signals backpressure with persistent queue disabled:

1. The Destination internal buffers fill up.
2. The event processors stop reading from Source buffers.
3. Source buffers fill up.
4. The Source signals backpressure to upstream senders (when supported).

TCP-based Sources. For TCP-based Sources such as Splunk TCP, Syslog over TCP, TCP JSON, and TCP Raw, the mechanism relies on TCP flow control, which is built into the protocol. When the Destination buffers fill, the event processor stops reading from the TCP socket buffer, the socket buffer fills, the TCP/IP stack notifies the sender to stop sending, and the sender pauses transmission until the socket buffer has capacity again. Because this happens at the individual socket level, the page calls out per-connection behavior:

• Different senders connected to the same Source may experience backpressure at different times.
• Senders transmitting higher volumes of data fill their socket buffers faster and experience backpressure sooner.
• Senders with slower network connections may have smaller effective buffer capacities and experience backpressure sooner.

Pull-based Sources. Pull-based Sources have built-in intelligence to detect backpressure, and when it is detected they stop or slow their data collection. The page's table of affected pull-based Sources covers: object storage Sources such as S3 and Azure Blob Storage, where file downloads pause; Database Collectors, where results streaming pauses; REST Collectors, where GET requests pause; Kafka, Confluent Cloud, and Amazon MSK Sources, where polling stops; Amazon Kinesis Data Streams, where stream reads pause; and Amazon SQS and Google Cloud Pub/Sub, where message retrieval pauses.

The page also documents side effects of pausing. If backpressure outlasts a file download from object storage, the retry can ingest the same data twice, and data purged or expired before recollection is lost. A paused Database Collector keeps its query active on the server, consuming CPU, memory, and connections, and can exhaust the connection pool. And when Kafka-based Sources pause polling long enough, the broker may treat the consumer as inactive and trigger a consumer group rebalance, disrupting other consumers and risking duplicate processing.

What happens to incoming data

Where the data itself ends up depends on the protocol behind the Source:

• TCP data is held on the sender side. When Cribl Stream stops acknowledging received data, the sender's outbound data queues in its local TCP buffer and transmission slows or pauses automatically. Once the Destination has capacity again and Cribl Stream resumes reading, the TCP stack resumes transmission and the sender's buffer drains.
• UDP data is dropped. UDP is a connectionless protocol, so UDP-based Sources such as UDP Raw and SNMP Trap do not have built-in backpressure support. When buffers are full, Cribl Stream drops incoming data, because there is no built-in way to signal the sender to slow down.
• Pulled data waits. For pull-based Sources, the pause in collection means files, query results, and messages remain at their origin until backpressure clears, which is where the download-timeout, connection-pool, and rebalance risks above land.

The page also documents what one blocked Destination does to the rest of a multi-Destination layout. Event cloning for multi-Destination routing happens after data passes through the Routes, so when a sender is blocked by one Destination, its data cannot flow through the Routes, the Clone Function or Output Router cannot create copies for other Destinations, and every Destination expecting data from that sender stops receiving it, even healthy ones. In the page's example, a Syslog Source feeding two Destinations via an Output Router stops delivering to the healthy one too, and the block persists until backpressure on the first resolves. The page's mitigation: set non-critical Destinations to Drop, or enable persistent queue.

Where persistent queues change the story

The persistent queue documentation describes the feature the backpressure page points to as a fix. Cribl Stream's persistent queues (PQ) supplement in-memory queues to minimize data loss during a backpressure event: when the event occurs, Cribl Stream temporarily stores data by writing it to persistent queue storage, and when the outage resolves, it forwards the data from the queue.

The page lists the situations persistent queues help minimize data loss in: a downstream Destination receiver is unreachable, a receiver is slow to respond, and a configured Source sends a higher volume of data than Cribl Stream can process.

Coverage is asymmetric. Persistent queues can be used on Streaming Destinations and Push Sources. Persistent queueing is not available for Pull and Collector Sources, which ingest data intermittently and/or from static files.

With persistent queue, the multi-Destination behavior changes where the backpressure page recommends it. If a Destination is configured to use persistent queue, Cribl Stream can queue that copy of the event locally while healthy Destinations continue to receive data, as long as the queue has available capacity. If a Destination is configured to Block and enters backpressure, Cribl Stream blocks delivery for that event across all Destinations that receive the same input data. And when a queue becomes full, Cribl Stream blocks or drops new data according to the configured Queue-full behavior for that Source or Destination.

For UDP specifically, persistent queue is strongly recommended, because UDP senders do not support backpressure signaling, as covered above. With source persistent queue enabled on a supported UDP Source, Cribl Stream buffers incoming events to persistent storage when downstream components are slow or blocked, and it bypasses the Max buffer size (events) setting in Advanced Settings, using only the persistent queue's own buffering controls. The page also suggests increasing the UDP socket buffer size (bytes) to handle brief traffic spikes.

The limits matter as much as the protection. Each queue has a finite size, so a system that cannot deliver data can eventually exhaust available disk space. Persistent queues operate at the Worker Process level, with each process maintaining its own queue. On Cribl.Cloud deployments, Cribl automatically allocates up to 1 GB of disk space per Destination, per Worker Process, for outbound uncompressed data, and when the queue fills up, Cribl Stream blocks data flow. Persistent queues provide limited protection against application failure, since a crash may lose in-memory data, and if you turn off persistent queue before the queues drain, you orphan the events still in the store. Writing to disk also demands CPU and adds a few milliseconds of latency per event, and the feature is only available for certain Cribl plans or license types.

Frequently asked questions

What does backpressure do in Cribl Stream when a Destination is full?

It propagates back through the Pipeline to your Sources, with the exact effect depending on the Source type. When persistent queue is not in use and the Backpressure behavior is set to Block, the event processors stop reading, Source buffers fill, and Sources that support backpressure signaling signal their senders while unsupported ones lose data. Propagation speed depends on buffer sizes, so a brief event might not reach all Sources, but sustained backpressure eventually impacts all connected senders.

How does Cribl Stream backpressure reach TCP-based Sources?

Through TCP flow control rather than a direct notification from the Destination. When the event processors stop reading, the TCP socket buffer fills, the TCP/IP stack notifies the sender to stop sending, and the sender's outbound data queues in its local buffer until the socket has capacity again. Because this happens at the individual socket level, senders attached to the same Source can hit backpressure at different times, with higher-volume senders hitting it sooner.

What happens to UDP data during backpressure in Cribl Stream?

Cribl Stream drops it. UDP is a connectionless protocol, so UDP-based Sources such as UDP Raw and SNMP Trap have no built-in backpressure support, and when buffers are full the incoming data is dropped rather than signaled back to the sender. The docs recommend source persistent queue on supported UDP Sources, which buffers events to persistent storage instead, and a larger UDP socket buffer size for brief traffic spikes.

How do persistent queues change backpressure behavior in Cribl Stream?

They let Cribl Stream store events on disk during the event instead of blocking the whole flow. A Destination configured for persistent queue can queue its copy of the event locally while healthy Destinations continue receiving, and supported Push Sources buffer incoming events instead of dropping them. Persistent queueing is not available for Pull and Collector Sources, each queue has a finite size, and a queue disabled before it drains orphans the events still in it.