High Availability and Failure Modes in Cribl Stream

This post covers the HA requirements from the two Cribl Stream pages, the Leader and Worker split, and the failure scenarios the pages name.

The HA requirements

The High Availability Requirements page opens with one instruction: before adding a standby Leader, fulfill all the requirements in the page. Both the primary and the standby Leader need enough disk space to hold the failover volume's contents (except the state directory), all configuration files, plus at least 10 GB in reserve for future use.

The Leader health check timeout gives the Leader time to pull all its configuration and the git repository from the failover volume; set it too low and the Leader can be terminated prematurely. Approximate recommendations: 2 minutes for smaller deployments, 5 minutes for more than 50 Worker Groups or Fleets, and 10 minutes for more than 100.

If you have a git timeout in local/cribl.yml, the page says make it equal to or greater than the system default of 10 minutes; all Leaders must have matching auth tokens.

Every Leader Node runs the latest NFS client (NFSv4 required), and the volume must offer at least 100 GB of space, IOPS of at least 200, ping or latency under 50 ms between Leaders and the NFS, and mtime updates. The HA requirements page suggests validating that latency with a tool like ioping: from the NFS mount, run:

ioping .

Mount options matter because the Leader touches large numbers of files whenever you use the UI or deploy configurations to Worker Nodes, and NFS's default behavior synchronizes access time updates for those files, often across availability zones and regions. Cribl recommends relatime (update the access time only if it is more than 24 hours ago, or if the file is being created or modified) or noatime, the most performant option, which never updates access times, hiding which files are accessed.

On AWS the page recommends Amazon Elastic File System, read/write access for the user, an EFS Throughput mode of Enhanced with Elastic, and Leader Nodes in the same region as the NFS.

Configure all Leaders behind a load balancer: port 4200 via a network load balancer, port 9000 via an application or network load balancer. By default the standby Leader returns HTTP 420 from GET /health, so a balancer that routes only to Leaders returning 200 sends client requests to the primary; UI and API requests that reach the standby on port 9000 are proxied to it.

Active-Active (Proxy) Mode lets both Leaders report healthy (200), with the standby proxying incoming requests to the elected primary; the page says this preserves single-writer semantics because only the primary owns Leader state. Set connection_proxy.procs to a nonzero value on each Leader and allow Leader-to-Leader traffic. Health checks use GET /health, polled every 60 seconds to align with Cribl.Cloud's internal monitoring, and route traffic exclusively to Leaders that return 200.

Who does what: leaders and workers

The Set Up Leader and Worker Nodes page defines the split: each instance in a Distributed deployment runs as a Leader Node governing the deployment or a Worker Node managed by the Leader. Internally the Leader mode is called master, an earlier, outdated name, which is why master appears in commands, config keys, and environment variable names.

Mode is set with environment variables, UI settings, the CLI, or instance.yml; environment variables take priority. CRIBL_DIST_MODE defines the mode (worker or leader) and CRIBL_DIST_LEADER_URL the Leader Node URL. The leader and worker page shows both export lines, for a Worker and for a Leader that binds to 0.0.0.0:

export CRIBL_DIST_LEADER_URL=tcp://<authToken>@<leaderHostname>:4200 ./cribl start

export CRIBL_DIST_LEADER_URL=tcp://<authToken>@0.0.0.0:4200 ./cribl start

If CRIBL_DIST_LEADER_URL is set you can omit CRIBL_DIST_MODE and the mode defaults to worker; if any CRIBL_DIST_* variable is defined in the system, Distributed mode settings can no longer be configured via the UI.

The UI path is Settings, then Global, then Distributed Settings and General Settings, where Mode is Leader or Stream: Managed Worker (managed by Leader). Leader Settings covers the Leader's Address and Port, the Worker Address, and Forward to Leader API, on by default.

Teleporting (remote UI access) lets you inspect Workers from the Leader: enable Teleport on a Worker Group's row, then open a Worker from the Workers page, where a purple border marks the remote view. The workerRemoteAccess key in groups.yml is an alternative. Tags added in the teleport view help the Leader map the Worker to a Worker Group.

The CLI offers mode-master and mode-worker; mode-worker requires -H, -p, and -u parameters, and Cribl Stream will need to restart after the command is issued.

The distributed section of $CRIBL_HOME/local/_system/instance.yml also sets the mode per instance. The leader and worker page shows the Leader variant:

distributed:
  mode: master
  master:
    host: <IP or 0.0.0.0>
    port: 4200
    forwardToLeaderApi: true
    tls:
      disabled: true
    ipWhitelistRegex: /.&ast;/
    authToken: <authToken>
    enabledWorkerRemoteAccess: false
    compression: none
    connectionTimeout: 5000
    writeTimeout: 10000

Switching an instance from Single-instance to Distributed mode to work as the Leader takes an auth token securing Node-to-Leader communication; the page also points to a Cribl University course on attaching Workers to Leaders.

Failure modes the docs name

The pages name the scenarios below; where a page stops at the failure point, so does this section.

Health check timeout set too low. A too-low timeout can result in the Leader being terminated prematurely, since the Leader uses that window to pull configuration and the git repository from the failover volume.

NFS IOPS below the floor. IOPS below the required 200 can cause excessive latency.

Leaders and NFS far apart. If the Leader and the NFS storage are distant from each other, the issues the page lists are latency in UI and/or API access, missing metrics between Leader restarts, and slower performance on data Collectors.

Default NFS access-time synchronization. NFS's default behavior synchronizes access time updates for the files the Leader touches, often across availability zones and regions, which the page says can introduce problematic latency; relatime or noatime is the remedy.

Load balancer health checks polled too often. Polling more often than every 60 seconds, for example every 5 seconds, can overload the Leader's API Process.

Source-level health checks triggering throttling. Frequent requests to a Source-level health check endpoint can trigger throttling; Cribl then returns a 503 Service Unavailable response that the page warns may mean the checks run too often rather than an actual failure.

Local changes made over teleport. Changes made directly on a Worker are not propagated to the Leader, and the next commit and deploy from the Leader overwrites them; adding tags is the only exception, and the Preview view does not reflect such changes.

Tag changes remapping a Worker. Changing a Worker's tags can cause the Leader to map it into a new Group.

Distributed settings locked after HA failover. Once you configure Leader High Availability/Failover, further Distributed Settings changes are locked for each Leader Node; the page says update them by modifying configuration files instead.

What the docs flag

A few more caveats the pages flag:

• The failover volume is not the installation directory: install Cribl Stream on fast local disk on every Leader, since network filesystems (NFS, EFS, and similar) are not supported there, and NFS is for the failover volume only. A Leader in failover mode resolves $CRIBL_HOME to a path that depends on its replication state, so read the disk requirement against the installation directory.
• Health checks are supported only on port 9000, over https once TLS is configured and http otherwise.
• If you need to access the same target through an AWS Network Load Balancer, use an IP-based target group and deactivate client IP preservation.
• On Cribl.Cloud the Worker Groups page adds columns for Worker type and, for Cloud Groups, anticipated ingress rate and Provisioned status; tags can be added only to hybrid Worker Nodes, after accepting a confirmation modal.
• Source-level health check endpoints exist for many HTTP-based Sources, enabled in the Advanced Settings tab; a 200 OK response indicates the Source is healthy.

Frequently asked questions

How much disk space does a Cribl Stream Leader need for high availability?

Both the primary and the standby Leader need enough space for the failover volume's contents, except the state directory, all configuration files, and at least 10 GB in reserve. It applies to all Leaders before you add a standby.

Why do the Cribl Stream docs recommend relatime or noatime on the failover volume?

The Leader touches large numbers of files whenever you use the UI or deploy configurations to Worker Nodes, and NFS's default behavior synchronizes access time updates for those files, often across availability zones and regions. noatime is the most performant option, at the cost of not seeing which files are accessed.

What happens when you edit a Cribl Stream Worker directly through teleporting?

Cribl Stream does not propagate the changes to the Leader Node, and the next commit and deploy from the Leader overwrites them, with adding tags as the only exception. The Preview view in teleport does not reflect such local changes either.

How do high availability Leaders in Cribl Stream share a load balancer by default?

By default the standby Leader returns a standby status, HTTP 420, from GET /health, so a balancer that routes only to Leaders returning 200 sends client requests to the primary. Active-Active (Proxy) Mode makes both Leaders report healthy, with the standby proxying incoming requests to the elected primary.