Ingesting NetFlow and IPFIX Telemetry with Cribl Stream

NetFlow and IPFIX, briefly

The NetFlow & IPFIX Source page in the Cribl Stream docs describes a Source that supports receiving NetFlow v5, v9, and IPFIX (v10) data via UDP. Its header block states three constraints up front: the type is Push, it has no TLS support, and it has no Event Breaker support. It ingests NetFlow records like events from other upstream senders: fields broken out, a message header per record.

The page keeps the three versions on one ingestion path but handles their fields separately: v5 gets its own field map, v9 takes its field definitions mostly from Cisco NetFlow documentation, and IPFIX (v10) shares its field representation with v9 while its fields come from the IANA IPFIX Information Elements list, predefined in ipfix-information-elements.yml. The Source does minimal processing of the incoming UDP messages to stay consistent with the internal Cribl event model. To render the data as metrics, the page points to a pre-processing Pipeline or a Route.

Setting up the source

Configuration happens in the New Source modal, per the NetFlow & IPFIX Source page. Add the Source via QuickConnect (Routing > QuickConnect > Add Source) or Routes (Data > Sources > Add Source). General Settings:

• Input ID: a unique name to identify this Source.
• Description: optional.
• Address: where to listen, for example localhost, 0.0.0.0, or ::.
• Port: the port number.
• Enable pass-through: lets Stream forward events to a NetFlow Destination; when enabled, an additional event containing __netflowRaw routes the raw NetFlow packet data to a downstream NetFlow Destination, and these events do not count against ingest quotas.
• Version Support: which versions are processed; all versions (V5, V9, IPFIX) are enabled by default; disabling one treats its packets as unsupported messages.

The optional Tags setting filters and groups Sources in the Stream UI; tags are not added to events, and names are separated by a tab or hard return.

Persistent Queue Settings. Enable persistent queue defaults to off. Mode: Always On (default; events always go to the queue first) or Smart (engages only on detected backpressure). Smart is deprecated for hybrid and on-prem Source persistent queues and migrates automatically to Always On in January 2027; drain the queues before switching, per the page. The rest, with page defaults: Buffer size limit (bytes) (64KB to 10MB; default 1MB; replaces the deprecated event-based setting as of version 4.18.0), Commit frequency (42), File size limit (10 MB; must equal or exceed the buffer limit), Queue size limit (required; 5 GB; up to 1 TB), Queue file path ($CRIBL_HOME/state/queues plus a per-worker, per-input suffix), Compression (default None; Gzip available), and Queue-full behavior (Block, or Drop new data). On Cribl-managed Cloud Workers (Enterprise plan), only the toggle is exposed: a fixed 1 GB per PQ-enabled Source and Worker Process, uncompressed.

Processing Settings. Fields defines or modifies fields with JavaScript expressions, similar to the Eval function, in an advanced-mode modal with sample data. Pre-Processing selects a single existing Pipeline or Pack to run on this input before the Routes.

Advanced Settings. IP allowlist regex defaults to .* (all IPs); IP denylist regex defaults to ^$ and takes precedence. UDP socket buffer size (bytes) sets the SO_RCVBUF socket option (minimum 256, maximum 4294967295) and affects OS memory utilization. Template cache minutes only affects v9 flow records: how long, in minutes, NetFlow v9 templates are cached before discarding; shorter durations suit networks whose data patterns change often. Environment is a GitOps option: a single Git branch, or everywhere if empty.

Connected Destinations: Send to Routes enables conditional routing, filtering, and cloning via the Routing table; QuickConnect sends data to one or more Destinations via independent, direct connections. Then Save, and Commit & Deploy.

What the events look like

For both NetFlow v9 and IPFIX (v10), fields map to their most natural representation in Cribl: IPv4 addresses as dotted octet strings, IPv6 and MAC addresses as colon-separated strings, numbers of six bytes or less as numbers, and larger numbers as strings. Registered field attribute names are the camelCase versions of the SCREAMING_CASE names in the Cisco whitepaper, and unregistered fields arrive in an array of records with their type codes and byte buffers.

NetFlow v5 gets its own layout. General fields include _time (Unix time in seconds at receipt), source (in the form udp|<remote IP address>|<remote port>), host, inputInt (always set to zero), protocol (for example, TCP = 6; UDP = 17), and icmpType and icmpCode, only present if the protocol is ICMP. There is no _raw field, because the Source reads input data directly from bytes in a compact format. Flow statistics cover packets, octets, startTime, endTime, and durationMs (endTime minus startTime), and header carries subfields such as count, sysUptimeMs, flowSequence, samplingMode, samplingInterval, and version.

v9 Option Data Records report aggregate metadata about the exporters; they can include any of the data-record fields, plus a scopes field (types: system, interface, line-card, cache, template) and a natural-number value field. A Community ID can also be attached to a flow: connect the Source to a Route containing a Function that supports expression evaluation and call C.Net.communityIDv1(), its optional seed parameter supports different Community IDs per use case within the same flow.

What the docs flag

The deployment-specific caveats start with Cribl.Cloud. On Cribl-managed Worker Groups, UDP ingest is available only on ports 514 and 9514; the 20000 to 20010 range is TCP only, so the Source will not receive data there, and 514 and 9514 are also used by the default Syslog Source. Large UDP event rates can also make Cribl.Cloud drop data, per the page's UDP-protocol caveat; place hybrid Worker Nodes close to the sender and tune the OS UDP buffer. For any other UDP port, deploy a customer-managed (hybrid) Stream Worker Group.

On pass-through, the page is explicit. Cribl Stream can forward NetFlow packets to other NetFlow destinations but cannot modify the incoming packet, which is forwarded verbatim as it came in. It only routes packets from upstream exporters, cannot generate its own, and cannot forward non-NetFlow input data to NetFlow Destinations. It can forward NetFlow packets to non-NetFlow Destinations such as Splunk, Syslog, and S3, using the events without the __netflowRaw field, which NetFlow Destinations discard. It emits a decoded event per data and options data record, plus an undecoded whole-packet event when pass-through is enabled.

For multi-node setups, the Source uses the Leader Node to keep Worker Nodes in sync with NetFlow v9 and IPFIX template information. If the Leader is unavailable, Worker Nodes still process data with the templates they already have, but new or updated v9 templates are learned only by the Worker Node receiving that traffic, and the others drop any flowsets that use those new templates.

UDP tuning is the next flagged area. Incoming UDP traffic is buffered by the Linux kernel, and the default buffer may be too small at scale; the page calls a typical value of about 200 KB far too small for an even moderately busy syslog server. Check with sysctl net.core.rmem_max and the packet receive errors line of netstat -su; more than zero means lost events. Its live remedy, setting the kernel receive buffer to 25 MB:

$ sysctl -w net.core.rmem_max=26214400
net.core.rmem_max = 26214400
$ sysctl -w net.core.rmem_default=26214400
net.core.rmem_default = 26214400

The permanent change adds the same two lines to /etc/sysctl.conf, after which the page says to watch Worker process memory.

The last flag is a v9 data collision case. Cribl Stream identifies a template by Template ID, Source ID (unique to the upstream exporting device), and the IP address of the upstream exporting device. Collisions may occur when exporters are heterogeneous and their IP addresses are rewritten (such as by a proxy) before reaching Cribl Stream, so two exporters share an IP and send simultaneously. The workaround is unique IP addresses for the exporters, or the original ones.

Frequently asked questions

Which flow record formats does the NetFlow and IPFIX Source in Cribl Stream accept?

It accepts NetFlow v5, v9, and IPFIX (v10) data over UDP only. The page's header states the Source is a Push type with no TLS support and no Event Breaker support, and Version Support defaults to all versions enabled.

How are field values represented in NetFlow v9 and IPFIX (v10) records decoded by Cribl Stream?

IPv4 addresses become dotted octet strings, IPv6 and MAC addresses become colon-separated strings, and numbers of six bytes or less become numbers while larger numbers become strings. Registered field attribute names are the camelCase versions of the SCREAMING_CASE names in the Cisco whitepaper.

Which UDP ports can the NetFlow and IPFIX Source use in Cribl Stream on Cribl-managed Worker Groups?

On Cribl-managed Worker Groups in Cribl.Cloud, UDP ingest is available only on ports 514 and 9514, and the 20000 to 20010 range supports TCP only. Those ports are also used by the default Syslog Source, and the docs point to a customer-managed hybrid Worker Group for any other UDP port.

What happens to NetFlow v9 templates in Cribl Stream when the Leader Node is unavailable?

Worker Nodes keep processing with the templates they already have, but new or updated v9 templates are learned only by the Worker Node receiving that traffic. Other Worker Nodes do not get those updates and drop any flowsets that use the new templates.