OpenTelemetry (OTel) Ingestion in Cribl Stream

The OTel source

Cribl Stream's OTel Source receives trace, metric, and log events from senders compliant with the OpenTelemetry Protocol (OTLP). The OTel source documentation classifies it as a Push source with TLS support and no Event Breaker support, and lists both OTLP transports: gRPC, the Protocol default, and HTTP. It also accepts compressed inbound data (DEFLATE or gzip) and uncompressed data.

In General Settings you set an Input ID, an optional Description, the OTLP version (0.10.0 or 1.3.1, default 1.3.1), the Protocol (gRPC or HTTP), the listen address (default 0.0.0.0), and the Port (default 4317). OTel applications default to 4317 over gRPC and 4318 over HTTP, so change the Port for HTTP. Port 4318 is not available on Cribl-managed Worker Groups in Cribl.Cloud.

The source has no encoding setting; over HTTP, the encoding comes from the request Content-Type: application/x-protobuf gives Binary Protobuf, application/json gives JSON Protobuf. JSON Protobuf requires OTLP version 1.3.1, and with 0.10.0 the source rejects JSON requests with HTTP 501. Invalid JSON, malformed OTLP payloads, and unsupported HTTP paths return HTTP 400. JSON and binary payloads decode into the same internal events, so Routes and Pipelines behave the same either way.

The page borrows the OpenTelemetry Project's signal definitions: a trace is a tree of spans tracking a single request, a metric aggregates statistics into data points, and a log records an event. Extract spans, Extract metrics, and Extract logs are toggled off by default, making the source a pass-through with one event per incoming OTel payload, useful for forwarding complete OTel events onward. Enabled, they generate one event per span, data point, or log record. Extract logs is available only at OTLP version 1.3.1, and Cribl recommends enabling it to simplify log transformation.

Events carry three internal fields you can reference in filters, Routes, and Pipelines: __otlp.version (the OTLP version used to parse the event), __otlp.type (the telemetry type, such as logs, metrics, or traces), and __otlp.extracted (true for records extracted from a batch). They are not in final output unless configured at the Pipeline or Destination stage.

Under Auth methods the page lists Token, Token (secret), Basic, Basic (credentials secret), and OAuth, which validates bearer JWTs against signing keys from a JWKS endpoint. Multiple methods can be enabled; a request is accepted when its credentials match any enabled method, and sender credentials ride in an authorization key (HTTP header over HTTP, gRPC metadata over gRPC). The same page documents server-side TLS settings with optional mutual authentication.

OTLP 0.19.0 introduced breaking changes in field names, and the page warns that Routes or Pipelines built for OTLP 0.10.0 may break when used with OTLP 1.3.1. A warning box adds that to use configurations created before Cribl Stream 4.7 with OTLP 1.3.1, you must check their filters and logic for obsolete field names and modify them.

For troubleshooting, the page points to the Live Data and Logs tabs on the source's configuration modal and the Monitoring page. When an upstream OpenTelemetry Collector logs "Exporting failed. Will retry the request after interval", the fix is to raise Keep-alive timeout (seconds) in Advanced Settings to match the Collector's configuration.

Processing OTLP logs

The OTLP Logs Function page documents the transform: it ingests logs sent by the Cribl Stream OTel Source, normalizes (that is, formats) them, optionally batches them, and sends them on to the Cribl Stream OTel Destination. The events it processes can only come from the OTel Source, which always ingests events as OTLP logs protocol buffers.

The page walks the protobuf top-down: a log protobuf message contains one or more ResourceLogs objects, each holding one Resource and one or more ScopeLogs, each ScopeLogs holding one Scope and one or more LogRecords, and a LogRecord being the log entry. The source sends Cribl events with or without logs extracted, depending on its Extract logs setting.

With Extract logs off, the Function cannot batch the incoming events, even with Batch OTLP logs turned on; it cleans up the events and sends them on. Log events always pass through to the OTel Destination, and non-log events are sent through or dropped depending on the Drop non-log events setting.

With Extract logs on, the Cribl events are flatter: the resource and scope objects and the extracted records are all peers. When batching is enabled, each resource object gets one scope_logs object holding multiple scope and log_records pairs. Batching groups events that arrive within a time window and have identical OTLP Resource Attributes; the page's example of 1000 events from 100 servers in 30 seconds, with attributes identical except for the server each describes, produces a separate batch per server. Resource attributes are written once per batch, greatly reducing the data sent over the wire.

Its settings include a Filter expression (default true passes all events), a Final toggle that stops feeding downstream Functions, and Drop non-log events, off by default. Preserve native AnyValue wrappers, on by default, keeps body and attribute values already in OTLP AnyValue form, for example { "string_value": "..." }, instead of re-wrapping them; toggled off, names such as string_value can remain literal field names. Batch OTLP logs exposes Batch size (records after which a batch is sent regardless of timeout), Batch timeout (ms) defaulting to 200 milliseconds, Batch size limit (kb) where 0 is no limit, Batch log metadata keys (one batcher instance per distinct value combination), and a Metadata cardinality limit defaulting to 1000, beyond which events with a new combination are dropped.

On placement, the page says to run the Function before an OpenTelemetry Destination or a related OTel-based destination such as Google Cloud Observability, Dynatrace OTLP, or ServiceNow Cloud Observability, with Preserve native AnyValue wrappers enabled on both the Function and the destination, since one side alone can leave string_value as a literal field name. Cribl also recommends placing the Function last in the Pipeline.

The basic example uses the filter __inputId=='open_telemetry:open_telemetry' to select all events from an OTel Source with Extract logs off. The incoming event, copied from the OTLP Logs Function page:

{
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python"
        }
    },
    "scope_logs": [{
        "scope": {
            "name": "opentelemetry.sdk._logs._internal",
            "version": "unknown"
        },
        "log_records":[{
            "time_unix_nano": 1717010319837985500,
            "observed_time_unix_nano": 0,
            "body": "Recommendation service started, listening on port 9001",
            "dropped_attributes_count": 0,
            "severity_number": 9,
            "severity_text": "INFO"
        }]
    }],
    "_time": 1717010325.183
}

The resulting basic example outgoing event, copied from the OTLP Logs Function page:

{
    "resource": {
        "attributes": [{
            "key": "telemetry.sdk.language",
            "value": {
                "string_value": "python"
            }
        }]
    },
    "scope_logs": [{
        "scope": {
            "name": "opentelemetry.sdk._logs._internal",
            "version": "unknown",
            "attributes": []
        },
        "log_records":[{
            "time_unix_nano": 1717010319837985500,
            "observed_time_unix_nano": 0,
            "body": { "string_value": "Recommendation service started, listening on port 9001" },
            "attributes": [],
            "dropped_attributes_count": 0,
            "severity_number": 9,
            "severity_text": "INFO"
        }]
    }],
    "_time": 1717010325.183
}

The differences are where normalization lands: attribute values become key/value objects, the scope gains an empty attributes array, and the plain-string body becomes a string_value object, the shape that setting interacts with.

Putting it together

The two pages jointly document one chain: OTLP payloads arrive at the OTel Source over gRPC or HTTP, the source forwards one Cribl event per payload or individual extracted records, the OTLP Logs Function normalizes and optionally batches those events, and the Cribl Stream OTel Destination converts the result into OTLP log events per the Function page's basic example. The source page owns ingest, transports through TLS; the Function page owns the transform, normalization through handoff. The pages do not jointly describe a longer pipeline, so this post stops here.

What the docs flag

• Extract logs on the OTel Source is available only at OTLP version 1.3.1.
• With OTLP version 0.10.0, JSON requests return HTTP 501, as do other unsupported Content-Type values; invalid JSON, malformed OTLP payloads, and unsupported HTTP paths return HTTP 400.
• OTLP 0.19.0 changed field names, so Routes or Pipelines built for 0.10.0 may break under 1.3.1; the page's warning targets configurations created before Cribl Stream 4.7.
• The OTLP Logs Function takes events from the OTel Source only, and batches only when that source had Extract logs enabled.
• Port 4318 is not available on Cribl-managed Worker Groups in Cribl.Cloud.
• With an OTel Destination after the Function, enable Preserve native AnyValue wrappers on both sides to avoid literal string_value field names.

Frequently asked questions

Which protocols does the Cribl Stream OTel Source support?

The Cribl Stream OTel Source supports gRPC, the default, and HTTP. With HTTP, the encoding comes from the request Content-Type: application/x-protobuf is Binary Protobuf, application/json is JSON Protobuf.

When can the OTLP Logs Function in Cribl Stream batch OTLP logs?

The OTLP Logs Function can only batch when the sending OTel Source had Extract logs enabled. Batches go out on the configured Batch size, Batch size limit (kb), or Batch timeout (ms), which defaults to 200 milliseconds.

How does the OTLP Logs Function in Cribl Stream handle non-log events?

Non-log events are passed through or dropped depending on the Drop non-log events setting, which is toggled off by default. Log events, by contrast, always pass through to the OTel Destination.

What happens when a JSON request reaches a Cribl Stream OTel Source set to OTLP version 0.10.0?

The source rejects it with HTTP 501, because JSON Protobuf requires OTLP version 1.3.1. The 0.19.0 field-name change can also break Routes or Pipelines built for 0.10.0 when they run against 1.3.1.