Datadog with Cribl Stream

What it is for

Cribl Stream sends log, metric, and trace events to Datadog, which supports metrics of type gauge, counter, rate, and distribution via its REST API. Three primary Datadog endpoints must be accessible from your Cribl environment, copied below from the Datadog Destination page:

Logs: http-intake.logs.{domain}
Metrics: api.{domain}
APM Traces and APM Stats: trace.agent.{domain}

In each endpoint, the {domain} placeholder is the specific Datadog region you are using. It defaults to datadoghq.com for the US region, and you configure an alternative region with Datadog site under Optional Settings. The supported domains the Datadog Destination page lists are:

US: datadoghq.com (default)
US3: us3.datadoghq.com
US5: us5.datadoghq.com
Europe: datadoghq.eu
US1-FED: ddog-gov.com
AP1: ap1.datadoghq.com
Custom: custom.datadoghq.com

Configuring the destination

On the top bar, select Products, then Cribl Stream, and pick a Worker Group. Configure via QuickConnect (Routing > QuickConnect in Stream, Collect in Edge) or the Routes path (Data > Destinations), then select Add Destination.

The New Destination modal starts under General Settings with Output ID (a unique name identifying this Destination definition) and an optional Description.

Under Authentication, the Authentication method dropdown offers Manual (a field for an API key available in your Datadog profile), Secret (an API key (text secret) drop-down selecting a stored secret that references the API access token, with a Create link to store a new, reusable secret), and API key (enter your Datadog organization's API key).

The Optional Settings section documents:

• Datadog site: the region you are sending to. Defaults to US; other options are US3, US5, Europe, US1-FED, AP1, and Custom (manually enter a region URL not listed).
• Send logs as: content type for sending logs. Defaults to application/json (each log message is a JSON object); the text/plain alternative sends one message per line, with newline \n delimiters.
• Message field: the event field containing the message to send. If not specified, Cribl Stream sends a JSON representation of the whole event, regardless of whether Send logs as is set to JSON or plain text.
• Source, Host, and Service: names to send with logs. If you are sending logs as JSON objects, the event's source, host, and __service fields (if set) will override these values.
• Datadog tags: tags such as env:prod or env_staging:east, used in the default batching behavior controlled by Batch by tags; high tag cardinality can affect performance.
• Severity: default message severity; when sending logs as JSON objects, the event's __severity field (if set) overrides it.
• Allow API key from events: if on, any API key in the __agent_api_key internal field overrides the API key field's value. Useful when events originate from multiple Datadog Agent Sources, each configured with a different API key.
• Backpressure behavior: block, drop, or queue events when all receivers are exerting backpressure. Defaults to Block.
• Tags: filter and group Destinations on the Destinations page; not added to processed events. Separate (arbitrary) tag names with a tab or hard return.

When events also include the ddtags internal field, its value must be a single string of comma-separated key:value pairs (for example, env:dev, app:myapp), not an array. Cribl Stream merges UI tags with the event ddtags value, and event tags do not override the tags configured here.

Optionally adjust four further tabs, then select Save and Commit & Deploy. The Persistent Queue Settings tab displays when Backpressure behavior is set to Persistent Queue. Its Mode menu picks when the queue engages: Error (Destination unavailable or in an error state), Backpressure (when backpressure is detected, until it resolves), and Always On (immediately queue all events). A Worker/Edge Node that starts with an invalid Mode automatically switches to Error; this might happen on versions that do not support the other modes (older than 4.9.0). Also on this tab: File size limit (defaults to 10 MB), Queue size limit (defaults to 5 GB), and Backpressure duration limit (defaults to 30 seconds; displays only when Mode is Backpressure).

On Cribl-managed Cloud Workers (with an Enterprise plan), that tab exposes only Mode and Backpressure duration limit plus the Clear Persistent Queue button, and a maximum queue size of 1 GB disk space is automatically allocated per PQ-enabled Destination, per Worker Process. The limit is not configurable.

The Processing tab offers a Pipeline or Pack to process data before this output sends it, plus System fields added automatically to events that use the output, including cribl_pipe by default and supporting wildcards.

Under Retries, Honor Retry-After header honors a Retry-After header provided the delay is no longer than 180 seconds, and any received header takes precedence over all other Retries options. By default, 1xx and 3xx series are dropped, 408 and 429 are retried in memory or the persistent queue (if enabled), any other 4xx is dropped, and any 5xx is retried the same way. Settings for failed HTTP requests adds per-code retry entries such as Pre-backoff interval (ms) (defaults to 1000) and Backoff multiplier (default 2). Retry timed-out HTTP requests exposes the same backoff settings for timed-out requests.

Some of the documented Advanced Settings:

• Validate server certs: reject certificates not authorized by a CA in the CA certificate path, nor by another trusted CA.
• Compress: on by default and recommended; compresses the payload body before sending.
• Send counter metrics as 'count': unless toggled on, Datadog might transform counter metrics to gauge.
• Batch by tags: on by default, batches log events by API key plus Datadog tags; off groups solely by API key, reducing batch count and improving performance.
• Request timeout: seconds to wait before aborting a request; defaults to 30.
• Request concurrency: max concurrent requests per Worker Process; defaults to 5, minimum 1, maximum 32.

Using it

When this Destination receives APM trace or stats events, it forwards them to the trace.agent.{domain} endpoint: traces originally received by a Datadog Agent Source on /v0.3/traces, /v0.4/traces, /v0.5/traces, or /api/v0.2/traces go to Datadog's APM intake, and APM client stats received on /v0.6/stats or /api/v0.2/stats go to the stats intake. The Destination preserves the original payload format and routes events by each event's structure.

If an event contains the internal field __criblMetrics, Cribl Stream sends it to Datadog as a metric event, serializing the payload to match Datadog's Submit Metrics API format. Name, type, value, and dimensions come from __criblMetrics; the built-in _time field controls the outbound timestamp, and if _time is missing or not numeric, Cribl Stream uses the current time. All other events are sent as logs.

If you need to proxy HTTP/S requests, the page points to the System Proxy Configuration documentation.

The configuration modal has troubleshooting tabs: Live Data (Start Capture to view real-time events), Logs (delivery logs with errors and warnings), and Test (Run Test to verify the Destination is set up and reachable).

What the docs flag

• Send ddtags as a single comma-separated string: if you send it as an array (such as ["env:dev","app:myapp"]), Datadog rejects the payload.
• By default, a 429 (Too Many Requests) response not listed in the retry settings is treated as a non-retryable 4xx error and dropped; include 429 in Settings for failed HTTP requests to retry it.
• High cardinality in Datadog tags (many unique tag values) can result in excessive concurrent requests and potential 408 errors from Datadog; toggling Batch by tags off groups log events solely by API key.
• The deprecated Buffer size limit (events) setting was replaced as of version 4.18.0 by the byte-based Buffer size limit (bytes), which defaults to 1MB for upgraded Worker Groups and Fleets; update your configurations to the byte-based limit.
• Clear Persistent Queue (Cloud Enterprise only) permanently deletes queued data without delivering it, and the page says to use it with caution. It only appears after you define the Output ID.

Frequently asked questions

Which endpoints does the Cribl Stream Datadog destination send to?

It sends to three primary Datadog endpoints: one for logs (http-intake.logs.{domain}), one for metrics (api.{domain}), and one for APM Traces and APM Stats (trace.agent.{domain}). The {domain} placeholder is the Datadog region you are using, and it defaults to datadoghq.com for the US region.

What authentication methods does Cribl Stream support for a Datadog destination?

The Authentication section offers Manual, Secret, and API key methods. Manual and API key each take an API key that you enter directly, while Secret selects a stored secret that references the API access token.

What does the Cribl Stream Datadog destination send if I do not set a Message field?

Cribl Stream sends a JSON representation of the whole event. That applies regardless of whether Send logs as is set to JSON or plain text.

How does Cribl Stream decide which events become Datadog metrics?

If an event contains the internal field __criblMetrics, Cribl Stream sends it to Datadog as a metric event, and all other events are sent as logs. The metric name, type, value, and dimensions come from __criblMetrics, and the built-in _time field controls the outbound timestamp.