OS Tuning for Large Cribl Stream Deployments

The OS Tuning for Large Deployments page in the Cribl Stream docs covers the Linux kernel and operating system settings large on-prem deployments may need, and this post walks its four areas in order, with the page's triggers, values, and caveats.

When you need to tune

Per the page, large on-prem deployments that handle high connection volumes or data rates may require OS tuning to maintain optimal performance, and to prevent performance issues on high-load Worker Nodes, you might need to tune your OS to address common bottlenecks like connection tracking table exhaustion and buffer overruns.

The page attaches a trigger to each area. Connection tracking: nf_conntrack: table full, dropping packet in /var/log/messages or from journalctl -k, meaning you should increase the conntrack table size and potentially adjust related parameters. File descriptors: a too many open files error in cribl.log with an EMFILE or ENFILE code. UDP buffers: rising packet receive errors or RcvbufErrors. SELinux: a hardened environment running SELinux in enforcing mode.

The settings, one by one

Connection tracking (netfilter conntrack)

The section applies to Worker Nodes handling high-volume inbound network connections, where conntrack could become a bottleneck: enabled by default on most Linux distributions, it maintains a table of active network flows, and when the table fills up, the kernel drops packets, which can cause connection failures and data loss. The two sysctl parameters it lists:

• net.netfilter.nf_conntrack_max: maximum number of flows tracked concurrently; by default it derives from the buckets value: nf_conntrack_max= nf_conntrack_buckets × 4.
• net.netfilter.nf_conntrack_buckets: hash buckets for the conntrack table; if not set at module load, the kernel computes a default from total system memory (roughly total memory in bytes / 16384) with floor and ceiling limits applied.

First check the current max against the active count via sysctl or the corresponding procfs entries: if the count approaches or equals the max during normal operation, increase capacity. Method 1, the page's recommended approach, gives the table more capacity, set high enough to maintain adequate margin above your peak observed load; for very large deployments handling hundreds of thousands of concurrent connections, you may need values of 4 million or higher. The example, copied from the OS Tuning page:

# For example, to set approximately 2 million tracked connections (adjust per sizing)
echo "net.netfilter.nf_conntrack_max=2097152" >> /etc/sysctl.conf
sysctl -p

Adjust per your deployment's actual connection load, and if you substantially increase nf_conntrack_max, review nf_conntrack_buckets for balance: max defaults to approximately 4 times the bucket count, and the ratio keeps the hash table efficient.

Method 2 covers the extreme case: in extremely high-throughput, purpose-built environments where Cribl Stream is the only service and network overhead is a top concern, disabling conntrack entirely can be considered an aggressive measure. The page calls it often the simplest and most effective measure for some on-prem deployments, but says to approach it with caution; its side effects follow under What the docs flag.

File descriptor limits

The section applies to Leader and Worker Nodes. The Cribl Stream systemd unit sets LimitNOFILE=65536 by default, which aligns with the page's minimum recommendation. The limit applies per process, not systemwide or per user: the API Process and each Worker Process can each open up to 65,536 file handles, generally more than sufficient for high-volume workloads, including file-based Destinations, especially with high-cardinality partition expressions, and large numbers of concurrent inbound and outbound connections.

On diagnosis: the default value is typically sufficient for normal operation, and increasing the limit is generally not the correct fix, since it usually only postpones identifying the root cause of a process hitting its ceiling. The cribl.log file shows the error with one of two codes:

• EMFILE (Too many open files): the per-process limit has been reached; the common case that LimitNOFILE addresses.
• ENFILE (File table overflow): the systemwide kernel-level limit for all open files has been reached; the page puts the fix, increasing the fs.file-max sysctl, outside its scope as the error is rarely encountered.

The common EMFILE case is usually the API Process being overwhelmed because Worker Processes are too busy, since the API Process receives the connection attempt first, then hands the file descriptor to a Worker Process. It typically occurs when a large number of clients, on the order of thousands, hold connections open on excessively long timeouts, and the API Process accumulates waiting connections until it exhausts its descriptor limit. The proper fix is addressing Worker load via the Pipeline Profiling documentation.

If investigation confirms the workload legitimately requires more file handles, such as in extremely large, high-volume, or unique deployments, raise the limit via a systemd override. The sequence concatenates three consecutive snippets from the OS Tuning page (edit command, example content, restart), values verbatim:

# Create or edit the systemd override
sudo systemctl edit cribl
[Service]
LimitNOFILE=262144
sudo systemctl restart cribl

Verify the running limit via the /proc/<pid>/ limits file of any Cribl Stream process.

UDP receive buffers

The section applies to Worker and Edge Nodes, specifically Nodes hosting UDP input Sources. For high-rate UDP ingest (Syslog, NetFlow, IPFIX), the default kernel UDP receive buffers may be insufficient, leading to packet drops before data reaches Cribl Stream; increasing them can significantly reduce packet loss. Cribl recommends tuning UDP buffers to accommodate short-lived but large data spikes.

Before tuning, confirm the data is properly distributed across all Worker Nodes in the Group; if only a subset is hit, fix the distribution first, which might resolve the packet drops entirely, making buffer adjustments unnecessary. For consistent high volume, scaling your environment (adding worker processes or nodes) is the correct approach. Watch for drops on modern operating systems with ss -ua, or netstat -su if ss is unavailable.

Method 1 tunes at the kernel level, a system-wide OS configuration that applies the buffer increase to all UDP sockets on the system. The commands, copied from the OS Tuning page:

# Set to 25 MB (adjust based on your volume)
echo "net.core.rmem_max=26214400" >> /etc/sysctl.conf
echo "net.core.rmem_default=26214400" >> /etc/sysctl.conf
# Apply the changes immediately without rebooting:
sysctl -p

Method 2 sets the buffer per input in Cribl Stream configuration, which the page generally recommends: the setting is portable, becoming part of your configuration and applying to newly deployed Nodes automatically, and scoped, applying only to the UDP sockets of Stream Sources with the setting defined. Start with 25 MB and adjust based on your data rates and observed drop rates. For detailed UDP tuning guidance, the page links to the Syslog Source (UDP Tuning) and UDP (Raw) Source pages.

SELinux

The SELinux section applies to Leader, Worker, and Edge Nodes. In hardened environments running SELinux in enforcing mode, ensure proper SELinux contexts are set on Cribl directories so Stream operates correctly, using SELinux audit logs to troubleshoot denials and adjusting policies as needed; the page links to the SELinux Configuration documentation.

What the docs flag

The page opens with a warning box: the parameters are Linux kernel and operating system settings, not Cribl Stream configuration options, directly affecting how the OS handles network connections and system resources for high-throughput Cribl Stream workloads. Caveats:

• Disabling conntrack is not generally recommended and carries significant consequences: it breaks stateful firewalls, since most modern firewall rules (iptables, nftables) rely on conntrack for stateful filtering, and disabling it requires a complete rewrite of your rules to be stateless.
• It also breaks NAT and container networking: NAT and IP masquerading are heavily dependent on conntrack, and disabling it breaks networking for container technologies like Kubernetes and Docker.
• If you run Cribl Stream on Kubernetes, do not disable netfilter or conntrack; Kubernetes networking (including kube-proxy and network policies) depends on conntrack for proper operation, so tune the conntrack parameters as in Method 1 instead.
• The ENFILE fix, increasing the fs.file-max sysctl, is outside the page's scope, as it is rarely encountered.

On systems: the settings are Linux kernel and operating system parameters, and the page names no specific Linux distribution or kernel version.

Frequently asked questions

What does the Cribl Stream OS Tuning page say about Linux versions?

The OS Tuning page frames every parameter as a Linux kernel or operating system setting, not a Cribl Stream configuration option, and names no specific Linux distribution or kernel version. It targets large on-prem deployments with high connection volumes or data rates.

How large should nf_conntrack_max be in a Cribl Stream deployment?

The page gives no single value: set the maximum tracked flows high enough to maintain adequate margin above peak observed load; deployments of hundreds of thousands of concurrent connections may need 4 million or higher. Its example sets net.netfilter.nf_conntrack_max=2097152, approximately 2 million, adjusted per sizing.

What is the default file descriptor limit for Cribl Stream processes?

The Cribl Stream systemd unit sets LimitNOFILE=65536 by default, aligning with the page's minimum recommendation. The limit applies per process, not systemwide or per user, so the API Process and each Worker Process can each open up to 65,536 handles.

Can I disable netfilter conntrack on Kubernetes nodes running Cribl Stream?

No. If you run Cribl Stream on Kubernetes, the page says do not disable netfilter or conntrack, because Kubernetes networking (including kube-proxy and network policies) depends on conntrack. It directs you to tune the conntrack parameters instead.