Suppressing Duplicate and Repeating Events in Cribl Stream

The Suppress Function page in the Cribl Stream documentation states the job in one sentence: the Suppress Function either drops or tags specified events over a specified time period, based on evaluation of a key expression. This post walks through the variants that page documents, each setting the page exposes, the placement statements it makes, and the caveats the docs flag, all from that single page.

What suppress actually does

Two settings define the window over which suppression works. Number to allow is the number of events to allow per time period, and it defaults to 1. Suppression period (sec) is the number of seconds to suppress events after Number to allow events are received, and it defaults to 30. With those defaults, each distinct key admits one event and then sits suppressed for the length of the period. The page's first example states the outcome for exactly that configuration as one event per unique host value allowed in every 30s.

The page documents two shapes of key expression, and the two shapes behave differently when a field is missing. A single-field key names one field directly. Set to host, the Function suppresses by the value of the host field, and the page's result note says events without a host field will not be suppressed. The second shape builds a composite key from several fields in a template string: the page's own example, ${ip}:${port}, uses the ip and port fields from each event to generate the key, and ${host}:${port} appears in three of the page's four worked examples. Those are the only key expression forms the Suppress Function page documents.

It documents two handling modes for events inside a suppression window, selected by the Drop suppressed events toggle, which defaults on. Left on, suppressed events are omitted from logs as they exit the Function, and the next event allowed through gains a suppressCount: N field whose N indicates the number of events dropped in the preceding Suppression period. The page illustrates the field with a screenshot of an allowed event. Toggled off, no events are dropped; instead, all qualifying events gain an added field suppress=1, which can be used downstream to further transform those events.

The first example, copied from the Suppress Function page, suppresses by the value of the host field:

Filter: true
Key expression: host
Number to allow: 1
Suppression period (sec): 30

Run against a datagen sample source that generates at least 100 events over 2 minutes, the result is one event per unique host value allowed in every 30s, and events without a host field will not be suppressed.

The second example, copied from the Suppress Function page, keys on the tuple value of host and port:

Filter: true
Key expression: ${host}:${port}
Number to allow: 1
Suppression period (sec): 300

Its result is one event per unique host:port tuple value allowed in every 300s, and the page immediately flags that suppression will also apply to events without a host or a port field under this key.

The third example, copied from the Suppress Function page, guarantees that suppression applies only to events carrying both fields by gating the Function on a Filter that checks for their presence:

Filter: host && port!=null
Key expression: ${host}:${port}
Number to allow: 1
Suppression period (sec): 300

The fourth example from the Suppress Function page shows the decorate mode, with Drop suppressed events toggled off:

Filter: true
Key expression: ${host}:${port}
Number to allow: 1
Suppression period (sec): 300
Drop suppressed events: Toggle off

No events are suppressed under that configuration. Rather, all qualifying events gain the added suppress=1 field, which can be used downstream to further transform those events.

Setting it up

The Usage section of the Suppress Function page documents these settings:

• Filter: a JavaScript filter expression that selects the data fed through the Function. Defaults to true, which evaluates all events. The page points to the Filter Expressions page for further details on filter usage.
• Description: a simple description of the Function. Defaults to empty.
• Final: toggle on to stop feeding data to downstream Functions. Default is toggled off.
• Key expression: the suppression key expression used to uniquely identify the events to suppress.
• Number to allow: the number of events to allow per time period. Defaults to 1.
• Suppression period (sec): the number of seconds to suppress events after Number to allow events are received. Defaults to 30.
• Drop suppressed events: specifies whether suppressed events should be dropped, or just tagged with suppress=1. Defaults to toggled on, meaning drop.

Advanced Settings carries three cache-related settings, and the page attaches the same advice to each of them: contact Cribl Support to understand the implications before changing the default. Cache size limit is the maximum number of keys that can be cached before idle entries are removed, with a default of 50000. Suppression period timeout is the number of suppression periods of inactivity before a cache entry is considered idle; it defines a multiple of the Suppression period (sec) value and defaults to 2. Num events to trigger cache clean-up checks the cache for idle sessions every N events when the cache size exceeds the Cache size limit, and its default is 10000.

Where it sits in the pipeline

The placement guidance on the page is narrow. An info box states that each Worker Process executes this Function independently on its share of events, pointing to the Functions documentation for the shared-nothing architecture behind that statement, so each Worker Process makes the drop-or-tag decision for the events it receives.

The only ordering lever the page documents is the Final setting: toggled on, it stops feeding data to downstream Functions, and the default is toggled off. Beyond that, the page does not document where the Suppress Function belongs relative to other Functions in a pipeline.

What the docs flag

The page carries four caveats worth planning around:

• Template-string keys catch events with missing fields. The page's warning box states that suppression will also apply to events without a host or a port field, because if a field is not present, ${field} results in the literal undefined. That contrasts with the single-field example, where events without a host field will not be suppressed.
• Gate on field presence to work around it. The third example is the fix the page offers: the Filter host && port!=null guarantees that suppression applies only to events carrying both fields.
• Contact Cribl Support before touching the cache defaults. The page repeats that advice for the 50000 cache size limit, the 2 suppression period timeout, and the 10000 cache clean-up trigger.
• Suppression runs per Worker Process. Each process executes the Function independently on its share of events, per the page's info box, which defers to the shared-nothing architecture documentation.

The page closes by pointing to Cribl's blog post Streaming Data Deduplication with Cribl for further use cases and to the Filter Expressions page for writing the filters its examples rely on.

Frequently asked questions

What does the Suppress Function in Cribl Stream do with events that match its key expression?

It either drops or tags specified events over a specified time period, based on evaluation of a key expression. With Drop suppressed events left on, the suppressed events are omitted from logs as they exit the Function, and the next event allowed through carries a suppressCount: N field. With the toggle off, no events are dropped, and all qualifying events gain a suppress=1 field.

How does the key expression shape change which events the Cribl Stream Suppress Function suppresses?

A key expression that names a single field, such as host, suppresses events by that field's value, and events that lack the field are not suppressed. A template string such as ${host}:${port} builds a composite key from multiple fields, and events without a host or a port field are suppressed as well, because a missing field evaluates to the literal undefined.

What settings does the Cribl Stream documentation expose on the Suppress Function?

The page documents Filter, Description, Final, Key expression, Number to allow, Suppression period (sec), and Drop suppressed events, with Filter defaulting to true, Description empty, Final off, Number to allow 1, Suppression period (sec) 30, and Drop suppressed events on. Advanced Settings adds Cache size limit, Suppression period timeout, and Num events to trigger cache clean-up, all three of which the page advises contacting Cribl Support about before changing.

Why does the Cribl Stream Suppress Function suppress events that lack the fields in a template key expression?

Because if a field is not present, ${field} results in the literal undefined, so events without a host or a port field are still suppressed under a template key. The fix the page offers is a Function-level Filter such as host && port!=null, which guarantees that suppression applies only to events carrying both fields.