Google BigQuery with Cribl Stream

What it is for

The Cribl Stream Google BigQuery destination writes your data out to Google BigQuery through the BigQuery Storage Write API. The docs say that eliminates the need to stage data in Google Cloud Storage first, reducing end-to-end latency. A callout on the page classifies the destination as Type: Streaming, with TLS support and persistent queue support both marked yes.

Configuring the destination

You configure the destination after selecting the Worker Group it should belong to. On the top bar, select Products, then Cribl Stream, and under Worker Groups select a Worker Group. Then either navigate to Routing > QuickConnect (or Collect on Edge), select Add Destination, pick the destination, and choose Select Existing or Add New; or use the Routes page, select Data > Destinations (More > Destinations on Edge), pick the destination, and select Add Destination.

In the New Destination modal, the General Settings you fill in for this destination are, per the BigQuery destination page:

Output ID: Enter a unique name to identify this BigQuery destination. If you clone this Destination, Cribl Stream will add `-CLONE` to the original Output ID.
Description: Optionally, enter a description.
Project ID: The Google Cloud project that contains the target BigQuery dataset.
Dataset ID: The BigQuery dataset to write to.
Table ID: The BigQuery table to write to. The table must already exist; Cribl Stream does not create tables automatically.

The remaining Optional Settings are Backpressure behavior, which chooses whether to block, drop, or queue events when all receivers are exerting backpressure and defaults to Block, and Tags, which filter and group destinations on the Destinations page and are not added to processed events.

Authentication

The Authentication method drop-down offers two options. Auto uses Google Application Default Credentials with no manual configuration, checking credential sources in order: first the GOOGLE_APPLICATION_CREDENTIALS environment variable, which should point to a service account key file or contain the credentials JSON directly; then credentials attached to the Google Cloud instance, such as workload identity federation or the instance metadata server. The page says to use Auto when running Cribl Stream on Google Cloud infrastructure or when credentials are managed externally. Secret selects a stored text secret containing the service account credentials JSON, and a Create link stores a new, reusable secret.

The Manual option, entering credentials inline, is not available for this destination. The page tells you to store service account credentials as a secret and use the Secret option instead.

Persistent queue

The Persistent Queue Settings tab displays when Backpressure behavior in General settings is set to Persistent Queue, and it buffers and preserves incoming events when the downstream destination has an outage or experiences backpressure. The Mode menu chooses when the queue engages: Error queues data to disk when the destination is unavailable or in an error state, Backpressure queues while backpressure lasts, and Always On queues and stores all data immediately, even when there is no backpressure. A page warning adds a condition: if a Worker or Edge Node starts with an invalid Mode setting, it automatically switches to Error mode, which might happen if the node is running a version that does not support other modes (older than 4.9.0) or encounters a nonexistent value in YAML configuration files.

The tab also sets storage behavior: a File size limit per queue file, a Queue size limit of disk space per Worker Process, a Queue file path, a Compression codec, a Queue-full behavior of block or drop new data, and a Backpressure duration limit that displays when Mode is set to Backpressure. The page scopes the tab differently on Cribl-managed Cloud Workers with an Enterprise plan: it exposes only the Mode and Backpressure duration limit settings, and some settings might not appear at all when the PQ storage mode is Network filesystem or AWS S3 at the Worker Group level.

Processing and advanced settings

Under Processing, the Post-Processing section lets you attach a Pipeline or Pack to process data before sending it out using this output, and adds System fields automatically to events that use this output; the field list supports wildcards. By default the list includes cribl_pipe, and the page's available fields include cribl_host, cribl_input, cribl_output, cribl_route, cribl_wp, cribl_group, and cribl_mode.

The Advanced Settings tab tunes how the writer batches events to BigQuery. These settings, with defaults as shown in the destination docs:

Queue size limit: Maximum number of batches that can queue before Cribl Stream applies backpressure. Defaults to `100`. Minimum: `1`.
Batch size limit (KB): Maximum size of a single append request sent to BigQuery. BigQuery enforces a 10 MB per-request limit. Defaults to `1024` KB. Minimum: `1`, Maximum: `10240`.
Flush period (sec): Maximum time to wait before sending a batch when the configured Batch size limit has not been reached. Defaults to `1` second. Minimum: `1`.
Concurrent request limit: Maximum number of in-progress API requests before Cribl Stream applies backpressure. Defaults to `5`. Minimum: `1`, Maximum: `100`.
Max send retries: Maximum number of retries per batch for retryable failures before the batch is dropped. Set to `0` to retry indefinitely. Defaults to `0`. Minimum: `0`.

Two more advanced settings matter here. Timestamp column names the BigQuery column that receives event timestamps: when set, Cribl Stream converts each event's _time field (Unix epoch, in seconds) to a BigQuery TIMESTAMP value at microsecond precision and writes it there. The column must exist in the table schema and be of type TIMESTAMP; the page says to use the setting when the table is partitioned by a timestamp column. And Environment, for GitOps use, can enable the configuration on a single Git branch; when empty, the config is enabled everywhere.

Using it

The page starts with delivery semantics. Cribl Stream writes to BigQuery using the BigQuery Storage Write API default write stream, which provides at-least-once delivery, so rows may be duplicated in the event of a retry.

For the event-to-row mapping, the page documents three behaviors. Each event field maps to a column in the target table, and all event fields are included as columns except those whose names start with __, the internal Cribl Stream metadata fields. If a Timestamp column is configured, the event's _time field goes to that column as a BigQuery TIMESTAMP; if the column name differs from _time, the _time field itself is omitted from the row. And the target table schema is fetched at initialization, so columns present in the event but absent from the table schema are silently dropped by BigQuery.

For permissions, your Google Cloud service account must have, at minimum, roles/bigquery.dataEditor on the target dataset. The page lists the role as granting bigquery.tables.updateData and bigquery.tables.get, which it calls the only permissions required by the Storage Write API.

For day-to-day use, the destination's configuration modal has three troubleshooting tabs. Live Data captures real-time events as they flow through the destination; Logs holds the delivery logs, including errors and warnings; and Test ensures the Destination is correctly set up and reachable, and clicking Run Test verifies that sample events are sent correctly. The page also points at the Monitoring page for a broader overview of data volume and rate.

What the docs flag

The page carries a handful of explicit caveats. The table you name in Table ID must already exist; Cribl Stream does not create BigQuery tables automatically. The Manual authentication option does not exist for this destination; the docs direct you to store credentials as a secret and use Secret instead.

Duplicate rows are the expected failure mode of at-least-once delivery on retry; the page's remedy is a deduplication key in your table design or query logic. Schema drift is silent: the table schema is fetched at initialization, so event columns missing from it are dropped by BigQuery without error. The timestamp mapping has a strict condition: the named column must exist and be of type TIMESTAMP, and if its name differs from _time, the _time field is omitted from the row. Finally, on Cribl-managed Cloud Workers with an Enterprise plan, the persistent queue tab is reduced to the Mode and Backpressure duration limit settings, and the queue size is not configurable there.

Frequently asked questions

Does the Cribl Stream BigQuery destination create the target table?

No, the table must already exist. Cribl Stream does not create BigQuery tables automatically, so you write the Project ID, Dataset ID, and Table ID of a table you have set up yourself.

Which authentication methods does the Cribl Stream BigQuery destination support?

It supports Auto, which uses Google Application Default Credentials with no manual configuration, and Secret, which selects a stored text secret containing the service account credentials JSON. The Manual option, which would enter credentials inline, is not available for this destination.

What delivery semantics does the Cribl Stream BigQuery destination provide?

It writes through the BigQuery Storage Write API default write stream, which provides at-least-once delivery. Rows may be duplicated in the event of a retry. If your downstream queries are sensitive to duplicates, the docs point to a deduplication key in your table design or query logic.

Which Google Cloud role does the service account need for the Cribl Stream BigQuery destination?

The service account needs at least roles/bigquery.dataEditor on the target dataset. That role grants bigquery.tables.updateData and bigquery.tables.get, which are the only permissions the Storage Write API requires.