New Relic with Cribl Stream

What it is for

Cribl Stream supports sending events to its New Relic Logs & Metrics Destination, which sends to the New Relic Log API and the New Relic Metric API. The docs list the Destination as Type Streaming, with TLS Support: Yes and PQ Support: Yes.

In release 3.1.2 and newer, the Destination authenticates using New Relic's Ingest License API key, because New Relic will retire the Insights Insert API keys it previously used for authentication.

Configuring the destination

From a Worker Group (select Products, then Cribl Stream, then a Worker Group), configure the Destination via QuickConnect (Routing, QuickConnect: Add Destination, Select Existing or Add New) or via the Routes (Data, Destinations: Add Destination).

General Settings take an Output ID, a unique name identifying this New Relic definition (cloning appends -CLONE to the original Output ID), an optional Description, and a Region: select a New Relic region endpoint, or select Custom to manually enter your New Relic region's URL if it's not listed.

Authentication offers an Authentication method dropdown with two options, per the New Relic Logs & Metrics Destination documentation:

• Manual: this default option exposes an API key field for your New Relic Ingest License API key, as you created or accessed it from New Relic's account drop-down.
• Secret: an API key (text secret) drop-down to select a stored secret that references the Ingest License API key, with a Create link to store a new, reusable secret.

If an incoming event contains the internal field __newRelic_apiKey, the Destination uses that field's value as the API key; otherwise it uses the API key you configured in the Authentication method settings.

Optional Settings: Log type, the name of the logType to send with events; it sets a default, and where a sourcetype is specified in an event, it overrides this value. Log message field names the field to send as the log message value; if not specified, the event is serialized and sent as JSON. Fields adds Name-Value pairs, the value being a JavaScript expression. Backpressure behavior selects whether to block, drop, or queue events when all receivers are exerting backpressure, and defaults to Block. Tags filter and group Destinations on the Destinations page; they are not added to processed events.

Optionally adjust the Persistent Queue, Processing, Retries, and Advanced settings, then select Save, then Commit & Deploy.

The Persistent Queue Settings tab displays when Backpressure behavior is set to Persistent Queue, buffering and preserving incoming events when a downstream Destination has an outage or experiences backpressure. On Cribl-managed Cloud Workers with an Enterprise plan, the tab exposes only Mode, Backpressure duration limit, and the destructive Clear Persistent Queue button: a maximum queue size of 1 GB disk space is automatically allocated per PQ-enabled Destination, per Worker Process; the limit is not configurable, and if the queue fills up, Cribl Stream blocks outbound data.

Mode selects when the persistent queue engages. The options, copied from the New Relic Logs & Metrics Destination page:

| Mode | Description |
| ---- | ----------- |
| **Error** | Queues and stores data on a disk when the Destination is unavailable or in an error state. |
| **Backpressure** | Queues and stores data to a disk when it detects backpressure from the Destination until the backpressure event resolves. |
| **Always On** | Cribl Stream immediately queues and stores all data on a disk for all events, even when there is no backpressure. |

• File size limit: maximum data volume per queue file before closing it; defaults to 10 MB, and on save must be at least the Buffer size limit (bytes).
• Queue size limit: maximum disk space per Worker Process; defaults to 5 GB, up to 1 TB unless a different Worker Process PQ size limit is configured.
• Buffer size limit (bytes): maximum memory buffered before flushing to disk; valid range 64KB to 10MB; defaults to 1MB.
• Backpressure duration limit: displays when Mode is Backpressure; seconds of backpressure before queueing; defaults to 30.

Processing Settings set the Pipeline or Pack that processes data before sending, and System fields added to events using this output (the default includes cribl_pipe).

Honor Retry-After header, when on, honors a Retry-After header (delay capped at 180 seconds) and takes precedence over all other Retries options; off ignores all such headers. Default Retry Behavior uses an exponential backoff algorithm for certain failed requests, overridable per error code. The default rules, copied from the New Relic Logs & Metrics Destination page:

| Status Code | Action |
| --- | --- |
| Any in the `1xx`, `3xx` series| Drop the request |
| `408`, `429` errors| Retry in memory or persistent queue if enabled, and initiate backpressure after limit is reached |
| Any other `4xx` series| Drop the request |
| Any `5xx` series| Retry in memory or persistent queue if enabled, and initiate backpressure after limit is reached |

A 429 not listed in the retry settings is treated as a non-retryable 4xx error and dropped. Each listed code sets a Pre-backoff interval (ms, default 1000), a Backoff multiplier (default 2), and a Backoff limit (ms, default and minimum 10,000, maximum 180,000).

Advanced Settings: Validate server certs rejects certificates not authorized by a CA in the CA certificate path nor by another trusted CA. Compress is on by default. Request timeout defaults to 30 seconds. Request concurrency defaults to 5, minimum 1, maximum 32, concurrent requests per Worker Process. Body size limit (KB) defaults to 1024 KB. Events-per-request limit defaults to 0, unlimited. Flush period (sec) defaults to 1. Failed request logging mode: None (default), Payload, or Payload + Headers, the last redacting all headers except declared Safe headers; sensitive headers such as authorization are always redacted.

Using it

Once you've configured log and/or metrics sources, create one or more Routes to send data to New Relic, per the New Relic Logs & Metrics Destination page.

Logs and metrics land in two different places in New Relic. To query logs, navigate to the Logs header option on the New Relic home screen and click the (+) button, then build queries with the Find logs where input field and add columns to the table view. To query metrics, click Browse Data, then Metrics, where you can search for metricNames and customize the time range and dimensions; alternatively, use NRQL to build your own query searches.

What the docs flag

• New Relic will retire the Insights Insert API keys this Destination previously used for authentication; in release 3.1.2 and newer it authenticates with the Ingest License API key.
• Where a sourcetype is specified in an event, it overrides the configured Log type value.
• If Log message field is not specified, the event is serialized and sent as JSON.
• By default, any 429 response not listed in the retry settings is treated as a non-retryable 4xx error and dropped.
• A Worker/Edge Node that starts with an invalid persistent queue Mode automatically switches to Error mode, for example on a version older than 4.9.0 or on a nonexistent YAML value.
• On Cribl-managed Cloud Workers with an Enterprise plan, the 1 GB maximum queue size per PQ-enabled Destination, per Worker Process, is not configurable, and a full queue blocks outbound data.
• The Buffer size limit (bytes) setting replaced the deprecated Buffer size limit (events) setting as of version 4.18.0, and the new byte-based limit defaults to 1MB for upgraded Worker Groups and Fleets.
• Use the Clear Persistent Queue button with caution to avoid data loss.

Frequently asked questions

How does the New Relic Logs and Metrics Destination in Cribl Stream authenticate?

The Manual method, the default, exposes an API key field for your New Relic Ingest License API key, and the Secret method offers an API key (text secret) drop-down with a Create link to store a new, reusable secret. In release 3.1.2 and newer, the Destination authenticates using the Ingest License API key, because New Relic will retire the Insights Insert API keys it previously used.

When does the Persistent Queue Settings tab appear, and what happens if the queue fills on Cloud Workers?

The tab displays when Backpressure behavior in General Settings is set to Persistent Queue, buffering and preserving incoming events when a downstream Destination has an outage or experiences backpressure. On Cribl-managed Cloud Workers with an Enterprise plan, a maximum queue size of 1 GB disk space is allocated per PQ-enabled Destination, per Worker Process; the limit is not configurable, and a full queue blocks outbound data.

Which HTTP status codes does the New Relic Destination retry by default?

It retries 408 and 429 errors and any 5xx series response, in memory or the persistent queue if enabled, and drops the 1xx and 3xx series and any other 4xx. By default, any 429 response not listed in Settings for failed HTTP requests is treated as a non-retryable 4xx error and dropped.

Where do logs and metrics land in New Relic, and how do you query each?

They land in two different places in New Relic. Logs are queried from the Logs header option on the home screen via the (+) button, the Find logs where input field, and the table view columns; metrics from Browse Data, then Metrics, searching for metricNames and customizing the time range and dimensions, or via NRQL.