Azure Event Hubs as a Cribl Stream Source

Cribl Stream receives data records from Azure Event Hubs using the Kafka protocol; the Azure Event Hubs Source page documents the configuration.

Event Hubs, briefly

The page lists Type Pull, TLS Support YES (secure API), and Event Breaker Support NO. Azure Event Hubs uses a binary protocol over TCP and does not support HTTP proxies, so Cribl Stream receives events directly from senders; firewall rules might need adjusting.

Reception works like normal Kafka: all Worker Nodes form one consumer group, each Worker takes its share of the load, and Workers poll every 5 seconds by default. In Cribl Stream 4.8.2 and newer, partition assignment is randomized and consuming Workers are picked at random, reducing uneven load when partitions are fewer than Worker Processes; random is not balanced, so even distribution of work is not guaranteed.

Setting up the source

On the top bar, select Products > Cribl Stream and pick a Worker Group (per the Azure Event Hubs Source page), then configure via QuickConnect (Routing > QuickConnect > Add Source, choosing Select Existing or Add New) or via Routes (Data > Sources, select the Source, then Add Source).

General and optional settings

In the New Source modal, Input ID uniquely names the source definition; cloning appends -CLONE. Description is optional. Brokers is the list of Event Hubs Kafka brokers to connect to, for example yourdomain.servicebus.windows.net:9093; the page says to get the hostname from the host portion of the primary or secondary connection string in Shared Access Policies, and an omitted port defaults to 9093. Event Hub name is the Event Hub (a.k.a. Kafka Topic) to subscribe to.

Group ID is the consumer group that includes this Cribl Stream instance, defaulting to Cribl. The page warns that each Group ID should be subscribed to only one Kafka Topic, the one in Event Hub name, since sharing one causes excessive Kafka rebalancing and reduced throughput; that it should not be $Default (in shared accounts that group might be subscribed to other Topics); that it should be unique for each of your Azure Event Hubs, Kafka, and Confluent Cloud Sources; and that a separate Source should be configured for each Group:Topic pair. From beginning reads from the earliest available data, applies only to initial subscription, and defaults on. Tags filter and group Sources in the Stream UI, are not added to processed events, and names separate by a tab or hard return.

Authentication

Authentication toggles on by default; when on, the section's remaining settings are all required. The SASL mechanism defaults to PLAIN, providing Basic Authentication that relies on Azure Event Hubs connection strings; selecting OAUTHBEARER enables OAuth Authentication via a different set of options.

Under Basic Authentication, Username should always be $ConnectionString. Authentication method: Manual (the default) exposes a Password field for the primary or secondary connection string from the Event Hubs workspace; Secret exposes a Password (text secret) drop-down over stored secrets, which can reside in Stream's internal secrets manager or, if enabled, an external KMS; a Create link adds a new secret.

The connection string takes this format, from the source page:

Endpoint=sb://<FQDN>/;SharedAccessKeyName=<your-shared-access-key-name>;SharedAccessKey=<your-shared-access-key-value>

OAuth authentication

With OAUTHBEARER, the options are Microsoft Entra ID based. The Microsoft Entra ID authentication endpoint defaults to https://login.microsoftonline.com, with https://login.microsoftonline.us and https://login.partner.microsoftonline.cn selectable instead. Client ID passes the client_id in the OAuth request parameter, Tenant identifier is your Microsoft Entra ID subscription's directory ID (tenant ID), and Scope takes the form https://<Event-Hubs-Namespace-Host-name>/.default. Authentication method: Manual (default, a Client secret field), Secret (a Client secret (text secret) drop-down with a Create link), and Certificate (a Certificate drop-down with a Create link).

TLS and processing

Use TLS defaults to toggled on; Validate server certs, also on by default, rejects certificates not authorized by a trusted CA, for example the system's CA.

Fields defines new fields or modifies existing ones with JavaScript expressions, similar to the Eval function: Field Name is a new field (unique within the event) or an existing one, and Value is an expression enclosed in quotes or backticks, possibly a constant. Auto parse under Pre-Processing detects each event's Datatype and extracts top-level fields from _raw before Source Fields and before the pre-processing Pipeline or Pack; off by default, and Fields values take precedence over same-named fields extracted from _raw. Pipeline selects an existing Pipeline or Pack to process the data before routing.

Retries

The default configuration starts with a 300 millisecond retry interval, doubles after each retry, and caps the maximum at 30 seconds. Retry limit defaults to 5 (0 means no retries). Initial retry interval (ms): default and minimum 300, maximum 600000. Backoff multiplier accepts 2 to 20 and drives the exponential backoff. Backoff limit (ms) caps the delay: default and minimum 30000, maximum 180000.

Advanced settings

Advanced Settings fine-tune the integration with Event Hubs Kafka brokers. The page links Azure Event Hubs' recommended configuration documentation; if unfamiliar with these parameters, contact Cribl Support about changing the defaults. Defaults per the page:

• Heartbeat interval (ms): 3000, lower than sessionTimeout, typically no more than 1/3 of it. Session timeout (ms): 30000, lower than rebalanceTimeout; a client missing heartbeats is removed, starting a rebalance. Rebalance timeout (ms): 60000; exceeded, the coordinator broker removes the worker.
• Connection timeout (ms): 10000, valid range 1000 to 3600000. Request timeout (ms): 60000. Authentication timeout (ms): 1000.
• Reauthentication threshold (ms): 10000; a small value with high latency might miss reauthentication before the broker closes the connection, a large one reauthenticates too soon, wasting bandwidth.
• Offset commit interval (ms) and offset commit threshold: both empty means offsets commit after each batch; both set means either condition commits.
• Byte limit, per partition: 1048576, must equal or exceed the server's maximum message size. Byte limit: 10485760 (10 MB).
• Error limit, per socket: consecutive request errors on one socket before the connection is discarded and reestablished, mitigating idle-connection issues; accepts 1 to 100, default 0 disables.
• Environment: with GitOps, enables the configuration on a single Git branch; empty means enabled everywhere.
• Minimize duplicates: one consumer per topic partition, reducing duplicates.

Page note: if you observe excessive group rebalances, or consumers not regularly pulling messages, increase Heartbeat interval, Session timeout, and Rebalance timeout.

Delivery, permissions, and internal fields

Send to Routes enables conditional routing, filtering, and cloning via the Routing table; QuickConnect sends the Source's data to one or more Destinations via independent, direct connections.

Cribl recommends a service principal with the minimum required permissions under Azure role-based access control (Azure RBAC), giving the Source a least-privilege dedicated identity. The minimum permission for Sources is Azure Event Hubs Data Receiver.

Cribl Stream also uses internal "meta" fields, not part of an event but accessible; Functions can use them for processing decisions. Fields for this Source:

• __inputId
• __topicIn (the Kafka topic the event came from)
• __partition
• When using Azure Schema Registry: __schemaId, __key, __headers, __keySchemaIdIn, __valueSchemaIdIn

What the docs flag

The main caveat is consumer group rebalancing: a state change on any Source (new config deployments, Worker Process adds or removals, crashes) affects all other Sources subscribed to different Topics in the same consumer group. The page's example: three Sources, three topics:

• Source_1 - Topic_1 - ConsumerGroup1
• Source_2 - Topic_2 - ConsumerGroup1
• Source_3 - Topic_3 - ConsumerGroup1

If Source 1 goes through a state change, Source 2 and 3 rebalance, and data flow stops until it completes. Deploying same-group Sources in one Worker Group shrinks side effects versus spreading them across Worker Groups; Bottom line: a unique Group ID for each Kafka, Confluent Cloud, and Azure Event Hubs Source.

For troubleshooting: the source modal's Live Data tab (Start Capture) shows real-time events, its Logs tab shows errors and warnings, and the Monitoring page graphs events and bytes in/out.

The page's common issue, from the Troubleshooting section:

KafkaJSProtocolError: Not authorized to access topics: [Topic authorization failed]

Its cause, per the page: the username does not have read permissions for the specified topic.

Frequently asked questions

How does Cribl Stream ingest from Azure Event Hubs?

Cribl Stream consumes from Azure Event Hubs over the Kafka protocol, as a pull source with TLS support. Worker Nodes form a consumer group, each taking its share of the load; Workers poll every 5 seconds by default.

Which SASL mechanisms does the Cribl Stream Azure Event Hubs source support?

Cribl Stream's source offers PLAIN, Basic Authentication via an Azure Event Hubs connection string, and OAUTHBEARER, OAuth Authentication via Microsoft Entra ID. For PLAIN the username is always $ConnectionString (Manual or Secret); for OAUTHBEARER the methods are Manual, Secret, or Certificate.

What is the minimum Azure permission for the Cribl Stream Azure Event Hubs source?

The minimum permission required for Sources is Azure Event Hubs Data Receiver. Cribl recommends a service principal with minimum required permissions under Azure role-based access control, on least privilege.

What does the page flag about consumer group rebalancing in Cribl Stream?

A state change on any Source in a consumer group affects all the others, and data flow stops until the rebalance completes. The page's remedies: a unique Group ID per Source, and one shared Worker Group for same-group Sources.