ClickHouse with Cribl Stream

What it is for

The Cribl Stream ClickHouse destination routes observability data, including logs, metrics, and traces, to ClickHouse for monitoring and analysis. Data is formatted and transmitted via HTTP requests, with both synchronous and asynchronous insert options. Authentication is Basic or SSL User Certificate, retry settings target reliable delivery, and field-to-column mapping is automatic or manual. The page's callout: Type: Streaming, TLS Support: Configurable, PQ Support: Yes. To get started, the page says, grab your ClickHouse instance's HTTP interface URL, database name, and table name, and add them to your Destination settings.

Configuring the destination

In Cribl Stream, select Products on the top bar, then Cribl Stream, and under Worker Groups select a Worker Group. Either navigate to Routing > QuickConnect (Stream) or Collect (Edge) and select Add Destination, or select Data > Destinations (Stream) or More > Destinations (Edge) and select Add Destination.

In the Destination modal, the General Settings are, per the ClickHouse destination page (a minimal rearrangement of the page's bullet list):

Output ID: Enter a unique name to identify this output definition. If you clone this Destination, Cribl Stream will add `-CLONE` to the original Output ID. A Description is optional.
URL: The URL of your ClickHouse instance's HTTP interface. This should include the protocol (http or https) and the port number if it's not the default. Example: `http://localhost:8123`.
ClickHouse database: Enter the name of the ClickHouse database where the target table resides. Ensure that the database exists and the authenticated user has the necessary permissions to write to it.
ClickHouse table: Enter the name of the target table. The name can contain letters (`A-Z`, `a-z`), numbers (`0-9`), and underscores `_`, and must start with either a letter or underscore. The table must exist and have a schema compatible with the data being sent.
Format: Choose the format for events sent to ClickHouse. Choose between:
Authentication type: ClickHouse supports several authentication methods for its HTTP interface.
Mapping: Determines how fields from Cribl Stream events are mapped to ClickHouse columns. Options include `Automatic` and `Custom`. See Event-to-Column Mapping for how each option works.
Async: The Async inserts toggle enables asynchronous inserts, allowing ClickHouse to buffer data in memory before writing it to disk. This can improve performance but may risk data loss if the server crashes before the data is written.

The Format trade: JSONCompactEachRowWithNames is optimized for efficiency by minimizing data size, while JSONEachRow is standard JSON with extra spacing that may be preferable when the time spent reformatting exceeds the cost of transmitting more data, for example when the ClickHouse instance runs on the same machine as Cribl. For Authentication type, None uses no authentication; Basic displays Username and Password fields for HTTP Basic credentials; Basic (credentials secret) selects a stored text secret holding the username and password, with a Create link for a new, reusable secret; and SSL User Certificate uses SSL certificates to verify the identity of the client.

With Automatic Mapping, Cribl Stream retrieves the table's column list with a DESCRIBE query when the Destination starts, matching each top-level event field to the column with the same name. It doesn't flatten nested fields, so the page says to use Custom for nested data such as OTLP logs. Exclude fields, a list of columns to exclude from the data sent, is available only with Automatic. With Custom, Retrieve table columns connects to a publicly accessible ClickHouse instance, authenticates with the provided credentials, and populates the Column Mapping table; each row takes a Column name, a Column type, and a Column value, which is a JavaScript expression. Wait for async inserts waits for the server to confirm the data is fully written. Backpressure behavior chooses Block, Drop, or Persistent Queue, and Tags group Destinations on the Manage Destinations page without being added to events.

TLS settings

Under TLS Settings (Client Side), Use TLS defaults to off. Server name (SNI) must be a host name, not an IP address, and Trusted server CA path names the PEM CA certificates on the Worker Node; the path can reference $ENV_VARS. For mutual authentication, a predefined Certificate auto-fills the certificate and private key paths, and the Passphrase decrypts the private key. TLSv1 or TLSv1.1 as Minimum TLS version shows a deprecation warning, and Maximum TLS version rejects connections above the threshold.

Persistent queue

The Persistent Queue Settings tab displays when Backpressure behavior in General settings is set to Persistent Queue, and it buffers and preserves incoming events when the Destination has an outage or experiences backpressure. The Mode menu chooses Error (queues while the Destination is unavailable or in an error state), Backpressure (queues until the event resolves), or Always On (queues all data immediately, even with no backpressure).

The page's defaults: File size limit 10 MB, which on save must be greater than or equal to Buffer size limit (bytes) or the combination is rejected; Queue size limit 5 GB, settable as high as 1 TB unless a different Worker Process PQ size limit is configured; Queue file path $CRIBL_HOME/state/queues, with /<worker-id>/<output-id> appended; Compression None with Gzip available; Buffer size limit (bytes) 1MB; and Backpressure duration limit 30 seconds, displayed only when Mode is Backpressure. Queue-full behavior is Block or Drop new data. Strict ordering is toggled on by default for FIFO forwarding, and toggling it off shows Drain rate limit (EPS), whose default 0 disables throttling. The byte-based buffer limit replaced the deprecated Buffer size limit (events) as of 4.18.0.

Some settings might not appear when the storage mode is Network filesystem or AWS S3 at the Worker Group level. On Cribl-managed Cloud Workers with an Enterprise plan, the tab exposes only Mode, Backpressure duration limit, and the destructive Clear Persistent Queue button, with a 1 GB maximum per PQ-enabled Destination, per Worker Process, that is not configurable; a full queue blocks outbound data, and a hybrid Group is the page's path to configuring queue size.

Processing and advanced settings

Processing Settings attaches a Pipeline or Pack under Post-Processing, and System fields are added automatically to events that use this output, as dimensions on metric events and labels on log events, with wildcard support. The list includes cribl_pipe by default, and the available fields also include cribl_host, cribl_input, cribl_output, cribl_route, cribl_wp, cribl_group, and cribl_mode.

Advanced Settings, with defaults: Validate server certs is toggled on; Compress is toggled on; Request timeout is 30 seconds; Request concurrency is 5 (minimum 1, maximum 32); Body size limit (KB) is 4096, settable as high as 25 MB (25600 KB); Events-per-request limit is 0 (unlimited); and Flush period (sec) is 1. Extra HTTP headers values are sent encrypted, Failed request logging mode is None (default), Payload, or Payload + Headers, Safe headers declares what may be logged in plaintext, Log last schema mismatch logs the most recent event that failed to match the table schema and points to the last-failed-format-event.json file, and Environment scopes a GitOps config to a single branch.

Retries

Honor Retry-After header honors the header, capping the delay at 180 seconds, and any received header takes precedence over the other Retries options. Destinations created before the 4.16.0 release do not support the default 4xx series retry behavior; if you would like that behavior, reconfigure the Destination. The default rules, rearranged from the table on the destination page:

Any in the `1xx`, `3xx` series: Drop the request
`401`, `403`, `408`, `429` errors: Retry in memory or persistent queue if enabled, and initiate backpressure after limit is reached
Any other `4xx` series: Drop the request
Any in the `5xx` series: Retry in memory or persistent queue if enabled, and initiate backpressure after limit is reached

Add Setting adds per-code behavior: an HTTP status code, a Pre-backoff interval (ms) defaulting to 1000, a Backoff multiplier defaulting to 2, and a Backoff limit (ms) defaulting to 10,000 with a maximum of 180,000. A 429 (Too Many Requests) not listed is treated as a non-retryable 4xx error and dropped, and Retry timed-out HTTP requests exposes the same three settings.

Using it

Under Automatic mapping, Cribl Stream runs a DESCRIBE query on the ClickHouse table when the Destination starts, and the results determine the set of columns to populate. For each event it sends the top-level field whose name exactly matches each column. Nested fields aren't flattened, and a field such as resource.attributes is never matched to a column. Unmatched event fields are not sent, and columns with no matching field receive no value, so ClickHouse applies the column's default; neither case logs a warning. Table changes are picked up on initialization only, so restart the Destination to pick up a column change. If the query fails or times out, the Destination shows an error status and doesn't send events until it succeeds; Cribl Stream retries the query periodically, and it honors Request timeout.

Custom mapping has no DESCRIBE dependency at startup, because it uses the columns in the Column Mapping table. For OTLP logs, set Mapping to Custom, then enable Extract logs on the OpenTelemetry Source, which generates one event per log record with body at the top level, plus attributes and severity_text when the record includes them. The destination page's column example, rearranged from its table:

Timestamp | DateTime64(9) | _time
ServiceName | LowCardinality(String) | resource.attributes['service.name']
SeverityText | LowCardinality(String) | severity_text
Body | String | body
LogAttributes | Map(LowCardinality(String), String) | attributes

A log record without attributes or a severity simply gets the column's default; Cribl Stream doesn't drop it. A present value that doesn't match the column type does cause a drop, since Body is a String column and LogAttributes is a Map(LowCardinality(String), String) column; the page's fix is to convert the values to strings first, for example with an Eval Function. In the configuration modal, Start Capture under Live Data shows real-time events, Logs holds delivery errors and warnings, and Run Test verifies sample events.

What the docs flag

If a DateTime or DateTime64 column, such as Timestamp, receives no value, ClickHouse applies the column's default, typically 1970-01-01 without a DEFAULT expression. With a TTL based on that column, ClickHouse can delete the rows immediately after insert; this is common with nested data such as OTLP, where the timestamp isn't a top-level field, and the page points to Custom mapping for such data.

Asynchronous inserts buffer data in memory, risking data loss if the server crashes before the write. A value that doesn't match the column type drops the entire event, in both mapping modes. And default 4xx series retry support is conditional: Destinations created before the 4.16.0 release do not support it; if you would like that behavior, reconfigure the Destination.

High Body size limit (KB) values can cause high memory usage per Worker Node, and the actual request body can exceed the limit. On Cribl-managed Cloud Workers with an Enterprise plan, the 1 GB queue maximum is not configurable.

Frequently asked questions

How does Automatic mapping work in the Cribl Stream ClickHouse destination?

Cribl Stream runs a DESCRIBE query on the target table when the Destination starts, and the results determine the set of columns to populate. For each event it sends only the top-level field whose name exactly matches a column name, and it does not flatten nested fields. Table changes are picked up on initialization only, so restart the Destination after changing the table's columns.

Which authentication types does the Cribl Stream ClickHouse destination support?

It supports None, Basic, Basic (credentials secret), and SSL User Certificate. Basic displays Username and Password fields for HTTP Basic credentials, Basic (credentials secret) selects a stored text secret holding the username and password, and SSL User Certificate uses SSL certificates to verify the identity of the client.

What happens when an event value does not match a column type in the Cribl Stream ClickHouse destination?

Cribl Stream drops the entire event, in both Automatic and Custom mapping. If you toggle on Log last schema mismatch, it logs the most recent event that failed to match the table schema and points to the last-failed-format-event.json file.

When does the Persistent Queue Settings tab appear on the Cribl Stream ClickHouse destination?

It displays when the Backpressure behavior option in General settings is set to Persistent Queue. The queue buffers and preserves incoming events when the Destination has an outage or experiences backpressure. On Cribl-managed Cloud Workers with an Enterprise plan, the tab exposes only Mode, Backpressure duration limit, and Clear Persistent Queue, and the 1 GB queue is not configurable.