Kafka In and Out of Cribl Stream: Confluent, MSK, and Kerberos

The two roles Cribl plays with Kafka

Cribl Stream sits on both sides of a Kafka topic. As a Kafka Source, it "supports receiving data records from a Kafka cluster," and every Worker Node joins the consumer group so Kafka manages each Node's data load. As a Kafka Destination, it "supports sending data to a Kafka topic." Both directions travel over "a binary protocol over TCP" and do not support HTTP proxies, and the docs note you might need firewall rule changes to allow the traffic. The reference pages document the exact controls below; the friction points are authentication choices and consumer group hygiene.

The Kafka source

General Settings in the New Source modal (rearranged from the Kafka Source page):

Input ID:          <your-input-id>
Description:       optional
Bootstrap servers: mykafkabroker:9092
Topic:             <your-topics>
Group ID:          <your-group-id>
From beginning:    on (default)

For Bootstrap servers, you "Specify the hostname and port (such as mykafkabroker:9092) or just the hostname (in which case Stream will assign port 9092)." For Topic, you "Press Enter/Return between multiple entries," although the page recommends subscribing each Kafka Source to only one topic to prevent excessive rebalancing, and suggests you "consider creating a dedicated Kafka Source for each one" otherwise. The optional settings include Group ID ("name of the consumer group to which this Cribl Stream instance belongs"), From beginning ("Whether to start reading from the earliest available data. Relevant only during initial subscription"), and Tags.

The rebalancing warning matters: the Group ID "should be unique for each of your Kafka, Confluent Cloud, and Azure Event Hubs Sources." A state change to any member of a consumer group (a config deploy, a Worker Process crash) makes the other members rebalance, stopping data flow until it completes.

Other Source behaviors documented on the page:

It "automatically detects compressed data in Gzip, Snappy, LZ4, or ZSTD format, and automatically decompresses the data upon ingesting it."
TLS is a toggle that defaults to off, with server validation, mutual authentication, and TLS version fields underneath.
SASL authentication offers the mechanisms PLAIN, SCRAM-256, SCRAM-512, and GSSAPI/Kerberos. The first three take credentials as Manual username and password entries or via a stored Credentials secret. Kerberos takes a Keytab location, a Principal (for example, kafka_user@example.com), and a Broker service class (for example, kafka).
OAuth adds Token URL, Client ID, and Client secret, plus optional parameters (such as scope for Amazon Cognito or audience for Auth0) and SASL extension fields; the page notes that "Confluent Cloud uses fields like logicalCluster or identityPoolId for granular access control."
The Schema Registry section handles Avro and JSON schemas stored in the Confluent Schema Registry, independently from bootstrap server authentication.
Leave the Server name (SNI) field in TLS Settings blank. The page warns that setting it "can cause traffic to be routed to the wrong brokers, because it interferes with the Kafka library's operation."

The Kafka destination

The New Destination modal's General Settings (rearranged from the Kafka Destination page):

Output ID:         <your-output-id>
Description:       optional
Bootstrap servers: localhost:9092
Topic:             <your-topic>

The Topic setting is "The topic on which to publish events. Can be overwritten using event's __topicOut field." That is the documented topic mapping: an event's __topicOut field can override the configured topic. The optional settings cover serialization and flow control:

Acknowledgments: "Select the number of required acknowledgments. Defaults to Leader."
Record data format: "Format to use to serialize events before writing to Kafka. Defaults to JSON." When set to Protobuf, the Protobuf Format Settings section appears, with Definition set OpenTelemetry and Object type Logs, Metrics, or Traces. The page is direct about conformance: "This Destination will drop non-conforming events."
Compression: a codec to compress data before sending, None, Gzip, Snappy, LZ4, or ZSTD. "Cribl strongly recommends enabling compression."
Backpressure behavior: block, drop, or queue incoming events when all receivers exert backpressure, defaulting to Block. Choosing the queue option exposes the Persistent Queue tab, whose modes are Error, Backpressure, and Always On.

Timestamp handling is easy to get wrong. "By default, when an incoming event reaches this Destination, the underlying Kafka JS library adds a timestamp field set to the current time at that moment." If you need the original creation time instead, use an Eval Function in a Pipeline to write it into a __kafkaTime field in Unix time; the page says "This Destination will recognize the __kafkaTime field and write its value into the timestamp field." Authentication mirrors the Source: same SASL mechanisms, same Kerberos fields, plus OAuth via the OAuth 2.0 client credentials grant, presented to the broker "with the SASL OAUTHBEARER mechanism." The same blank-SNI warning applies here.

AWS MSK differences

The Amazon MSK Source page is built on the same foundation but changes the identity story. General Settings add a Region field: "Select the name of the AWS Region where your Amazon MSK cluster is located." For the TLS section, the page notes that, for Amazon MSK Sources and Destinations, "IAM is the only type of authentication that Cribl Stream supports," and "Because IAM auth requires TLS, TLS is automatically enabled." The authentication section is an AWS credential story, not a SASL story:

The Authentication method dropdown offers three options:

Auto (default): uses the AWS SDK to obtain credentials in order: environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, IAM Identity Center (SSO), the shared credentials file, IAM roles for EC2 or ECS, a JSON file on disk, or other credential provider classes. "The Auto method works both when running on AWS and in other environments where the necessary credentials are available through one of the above methods."
Manual: static IAM credentials (Access key and Secret key) for Workers not in an AWS VPC, such as a private cloud.
Secret: a stored secret key pair from Cribl Stream's secrets manager, for the same off-AWS situations.

One caveat the page prints in a warning box: Azure-based Cribl.Cloud Worker Groups "do not have AWS credentials, so Auto finds none and authentication fails," so on an Azure-based Worker Group you select another method and supply credentials yourself. There is also an Assume Role section: "Enable for MSK" toggles Assume Role credentials for the cluster, with fields for the AssumeRole ARN, an optional External ID for third-party delegation, and a session Duration in seconds.

Kerberos without tears

The Kafka Authentication with Kerberos page walks through configuring a Kafka cluster that requires Kerberos, and ends with verification steps. Availability first: "Kerberos is available only on Linux, so you'll have to use a Linux host or Linux container to work through this procedure." And scope-wise: "Kerberos is not supported for Cribl-managed Cribl.Cloud Workers, but it is enabled for customer-managed (hybrid or on-prem) Worker Groups, whether the Leader is based in Cribl.Cloud or on-prem."

Once your Kafka instance runs with Kerberos, the page lists three steps on the host where Cribl Stream runs:

Install krb5-user (or the equivalent for non-Ubuntu systems) on your Worker Node.
Ensure a valid /etc/krb5.conf. The page notes Kerberos authentication with Kafka "specifically requires rdns = false in this config, but no other special settings," and Cribl Stream supports the file and directory cache types.
Ensure the keytab path in your configuration points to a valid keytab on each Worker Node.

The Destination configuration (values as shown on the page):

Brokers:              broker.kerberos-demo.local:9092
Topic:                Cribl

Enabled:              on
SASL mechanism:       GSSAPI/Kerberos
Keytab location:      /<path-to-client-keytab>/kafka-client.key
Principal:            kafka_producer@TEST.CONFLUENT.IO
Broker service class: kafka

The Source configuration is the same, plus a consumer group:

Brokers:              broker.kerberos-demo.local:9092
Topic:                Cribl
Group ID:             Cribl

Enabled:              on
SASL mechanism:       GSSAPI/Kerberos
Keytab location:      /<path-to-client-keytab>/kafka-client.key
Principal:            kafka_producer@TEST.CONFLUENT.IO
Broker service class: kafka

Two recurring details cause most of the breakage. The keytab path "must be available on all Worker Nodes," not just one. And the SASL mechanism dropdown label is GSSAPI/Kerberos, not just "Kerberos," on both the Source and the Destination.

Verification steps

The Kerberos page closes with a three-step verification:

Open the Source's Live Data tab and start a capture with Capture Time (sec) and Capture Up to N Events "set to high values (for example, 1000).
In a separate browser tab, navigate to the Destination modal's Test tab and select Run Test a few times.
Return to the Source's Live Data tab and "verify that the Source is capturing events."

The reference pages point at the same troubleshooting tabs: the Logs tab, which holds "detailed information about the ingestion process, including any errors or warnings that may have occurred," and the Monitoring page for data volume and rate. One common issue the Source page documents: the error KafkaJSProtocolError: Not authorized to access topics: [Topic authorization failed] means "The username does not have read permissions for the specified topic." If your capture stays empty, check topic permissions first.

Frequently asked questions

How does Cribl Stream read from a Kafka topic?

Every Worker Node joins the consumer group, and Kafka manages each Node's share of the data. The Group ID should be unique per Source, since a state change in any member forces a rebalance that stops data flow until it completes.

Which SASL mechanisms can the Cribl Stream Kafka Source use?

The page lists PLAIN, SCRAM-256, SCRAM-512, and GSSAPI/Kerberos. Kerberos takes a Keytab location, a Principal, and a Broker service class.

How is authentication different on the Cribl Stream Amazon MSK Source?

For MSK, IAM is the only authentication Cribl Stream supports, and TLS is enabled automatically because IAM requires it. Auto fails on an Azure-based Cribl.Cloud Worker Group, which has no AWS credentials to find.

What two details cause most Kerberos breakage in Cribl Stream?

The keytab path must be available on all Worker Nodes, not just one. The SASL mechanism dropdown label is GSSAPI/Kerberos on both the Source and the Destination.

Verified against Cribl Stream 4.20 documentation on October 1, 2026.