Prometheus Remote Write into Cribl Stream

Prometheus can hand its metrics to Cribl Stream either by pushing batches over the remote write protocol or by letting Stream pull them from the /metrics endpoints of the targets you list. This post covers the two Stream documentation pages behind those Sources: what each one accepts, the settings that matter on the way in, and the caveats the docs flag.

Two Prometheus-shaped sources

Start with the shape each page documents, because the configuration flow that follows differs for each Source.

The Prometheus Remote Write Source page says Cribl Stream supports receiving metric data from Prometheus instances that are configured to send data via the remote write protocol. The page marks the Source as Push, with TLS support and no Event Breaker support, and adds that it assumes incoming data is snappy-compressed.

The Prometheus Scraper Source page says Cribl Stream supports receiving batched data from Prometheus targets. That page marks the Source as Pull, with configurable TLS support and no Event Breaker support, and it links to the remote write page when you need streaming data instead of batched pulls.

Remote write target

Create the Source from the Cribl Stream UI: on the top bar, select Products, then Cribl Stream, pick a Worker Group, and add the Source either via QuickConnect under Routing or from Data > Sources with Add Source.

Under General Settings, the page documents an Enabled toggle, a unique Input ID (a clone gets -CLONE appended to the original ID), an optional Description, an Address to listen on that defaults to 0.0.0.0, a Port, and the Remote Write API endpoint, the absolute path on which Stream listens for Prometheus requests. The remote write page notes the endpoint defaults to /write and shows the resulting URL as:

http://<your-upstream-URL>:<your-port>/write

Optional Settings covers Tags, which you can use to filter and group Sources in the Cribl Stream UI; the page notes these tags are not added to processed events.

Authentication offers five options: None; Auth token, a bearer token that must ride in the HTTP authorization header, or select Generate for a new one; Auth token (text secret), a stored text secret; Basic, with Username and Password fields; and Basic (credentials secret), credentials referenced from a stored secret.

TLS is server-side and off by default. Select a predefined certificate or enter the certificate and private key paths by hand, both in PEM format and both able to reference $ENV_VARS, plus a passphrase to decrypt the key. Mutual authentication is optional and requires clients to present certificates, with client certificate validation on by default. Minimum and maximum TLS versions are also available, and selecting TLSv1 or TLSv1.1 as the minimum shows a deprecation warning.

Because Prometheus remote write requests do not specify metric types, the page documents fixed detection rules: a metric whose name ends in _total, _sum, _count, or _bucket is set to counter, and every other metric is set to gauge. The page notes this is consistent with the type detection practiced by other services that implement the remote write protocol. It also notes that Cribl Stream supports the timer type in addition to counter and gauge.

Useful Advanced Settings include a per-Source health check endpoint at /cribl_health that returns 200 while the Source is healthy, an Active request limit (on-prem only) that defaults to 256 per Worker Process with 0 for unlimited, a keep-alive timeout that defaults to 5 seconds, and IP allowlist and denylist regexes that default to .* and ^$.

The Prometheus scraper

A warning box on the page comes first: as of version 4.2, the Source is removed from Cribl Edge, and Edge users are pointed to the Edge-specific Prometheus Edge Scraper instead. What follows describes the Stream Source.

Under General Settings, the scraper page documents an optional Description, a unique Input ID, and a Discovery type drop-down with three mechanisms: Static (the default), DNS, and AWS EC2.

With Static discovery you enter the targets by hand in a Targets field, as full URLs or host[:port] entries. The Prometheus Scraper Source page lists these examples:

http://localhost:9090/metrics
localhost:9090
localhost

If you supply only host[:port], the page says the endpoint resolves to http://host[:port]/metrics: the protocol is prepended and the /metrics path is appended in all cases, even if you add a path. A nonstandard path requires the full URL, including protocol and path.

Optional Settings adds Extra dimensions (which default to host and source), Use field per metric (each metric name becomes the event field key, in the _metric : _value format), and Tags. Prometheus authentication has two methods: Manual, a Username and Password for Basic authentication against your targets, and Secret, a stored secret from Cribl Stream's internal secrets manager or, if enabled, an external KMS.

DNS discovery adds a DNS names field and a Record type drop-down that defaults to SRV (with A and AAAA as alternatives), plus a Target Discovery tab with a metrics protocol setting (http by default, or https) and a metrics path that defaults to /metrics.

AWS EC2 discovery adds a Region field, an Assume Role tab (an ARN, an External ID, and a session duration that defaults to 3600 seconds, minimum 900 and maximum 43200), and a Target Discovery tab whose Authentication method is Auto, Manual, or Secret. From there you can set the metrics port (default 9090), the metrics path (default /metrics), toggle Use public IP for discovered targets, and add Search filter rows. The page notes a filter row with no values returns only running instances.

The page explains that Stream pulls data using scheduled Collection jobs, with the Poll interval entry setting the schedule. With DNS or AWS EC2 discovery, each job has a Discover phase on a single Worker that returns one collection task per discovered target, and the job scheduler spreads those tasks across all available Workers.

What the docs flag

A few things the pages call out explicitly:

• The scraper Source does not support Prometheus metadata, per an info box on the page.
• On the remote write page, Smart mode for the Source persistent queue is deprecated on hybrid and on-prem deployments. The deprecation notice says affected configurations automatically migrate to Always On mode in January 2027, and it advises monitoring queue usage, draining the queues, and verifying routing in a test environment before changing anything.
• On Cribl-managed Cloud Workers (with an Enterprise plan), the persistent queue tab exposes only the Enable persistent queue toggle. Enabled, PQ runs in Always On mode with a maximum queue size of 1 GB of disk space per Source, per Worker Process, and that uncompressed limit is not configurable; the page points to a hybrid Group for configurable queue size, compression, and mode.
• Enabling Capture request headers on the remote write Source writes inbound headers into the __headers field, and the page warns the field can silently conflict with Destinations that also read it. HTTP Destinations such as Webhook, Loki, Microsoft Sentinel, Dynatrace HTTP, and Cortex XSIAM use it for dynamic request headers, and Kafka Destinations use it for message headers. Events can leave Stream while data is rejected downstream, and the page's stated fix is an Eval Function that removes __headers before the Destination.
• Both pages carry the same troubleshooting note: a request dropped because the token is invalid, and the pages note the message is logged only at the debug level.

Frequently asked questions

How does Cribl Stream receive Prometheus remote write data?

The Prometheus Remote Write Source in Cribl Stream is a Push Source that listens on an address and port you choose, with the remote write API endpoint defaulting to /write. It assumes incoming data is snappy-compressed, and the docs mark it as TLS-supported. Cribl Stream detects each metric's type as the data arrives.

Which target discovery types does the Cribl Stream Prometheus scraper support?

The scraper Source offers three discovery types: Static, the default, DNS, and AWS EC2. Static takes a hand-entered list of targets written as full URLs or host[:port] entries, while DNS resolves a list of DNS names using a record type that defaults to SRV. AWS EC2 discovers instances in a region, and a search filter row with no values returns only running instances.

How does Cribl Stream determine the type of a remote-written Prometheus metric?

A metric whose name ends in _total, _sum, _count, or _bucket is set to counter, and every other remote-written metric is set to gauge. The remote write page says this is consistent with the type detection practiced by other services that implement the remote write protocol. Cribl Stream also supports the timer type in addition to counter and gauge.

What persistent queue options does the remote write page document for Cribl-managed Cloud Workers?

On Cribl-managed Cloud Workers with an Enterprise plan, the persistent queue tab exposes only the Enable persistent queue toggle. Enabled, PQ is automatically configured in Always On mode with a maximum queue size of 1 GB of disk space per Source, per Worker Process, and that uncompressed limit is not configurable. The page points to a hybrid Group for configurable queue size, compression, and mode.