Cribl Outpost for Multi-Tenant Deployments

What an Outpost is

The Set Up Cribl Outpost page defines the component in one sentence: create a Cribl Outpost to relay communication between Worker Nodes and the Leader. Worker Nodes see an Outpost in the same way as a Leader Node, and connect with the Leader's auth token, since the Outpost has no separate auth token of its own.

Neither page uses the term multi-tenant, so this post sticks to the documented role: a relay between Worker Nodes and a Leader, organized by Outpost Groups. An Outpost Group defines the listener URL on which Worker Nodes connect to the Outpost, the TLS settings that secure that connection, and other options such as the retention time for disconnected Outpost Nodes.

Prerequisites, per the page:

• Cribl Outpost requires an Enterprise plan or Enterprise license.
• You can install Cribl Outpost on Linux only.
• Do not run an Outpost Node on the same host as a Stream Worker, so the two Nodes do not compete for resources.
• Create an authentication token on the Leader before deploying an Outpost Node.

Setting one up

Setup starts with the Outpost Group. Before you add an Outpost Node, revise the settings in the default_outpost Group, or create a new Outpost Group. Select Outposts in the sidebar, then Add Outpost Group, and enter a Name, an optional Description, and Tags. Leave Time to keep disconnected Outposts empty for the one-day default, then save.

Open Settings on the new Group. In Distributed Settings > Listener Settings, set Address and Port, which form the URL that Worker Nodes use to connect to Outposts in this Group. In Distributed Settings > TLS Settings, enable Use TLS to secure Outpost-Worker Node communication, providing certificate and private key information as needed. In versions older than 4.16, TLS information for Outposts was included in the individual Outpost Node settings instead of the Outpost Group, and on upgrade to 4.16, Outpost Nodes running older versions are placed in the default_outpost Group.

There are two ways of adding a new Outpost Node: the bootstrap script provided in the UI, or a binary file. For the bootstrap route, select Outpost in the sidebar, then an Outpost Group. Select Add/Update Outpost Node, then Add. Install package location defaults to Cribl CDN and can be changed to Download URL. The form also takes the authentication token, the target Outpost Group, the Leader hostname/IP (URI), the User and User Group to run Cribl as, and the Installation directory. Copy the resulting script, paste it onto the command line, and execute it. A successful add appears in the Outposts list.

For the binary route, go to the Cribl Download page, set the Software drop-down to Cribl Suite (Edge and Stream), select Download now to get the .tgz archive, and un-tar the install package:

cd /opt/
tar xvzf cribl-<version>-<build>-<arch>.tgz

Set the installation mode to mode-outpost to connect the Outpost to the designated Leader, using one of the authentication tokens:

/opt/cribl/bin/cribl mode-outpost -H <leader-hostname-or-IP> -p <port> -u <authToken> -S 1

The authToken must match an existing authentication token. -S 1 enables secure communication between the Outpost Node and the Leader via TLS; omit it and the connection will not use TLS, though TLS must be enabled to connect to a Leader in Cribl.Cloud. A new Outpost Node is placed in the default_outpost group by default; specify a different group with the -g parameter:

/opt/cribl/bin/cribl mode-outpost -H <leader-hostname-or-IP> -p <port> -u <token> -g <myGroup> -S 1

Finally, configure the installation as a service and start it:

sudo /opt/cribl/bin/cribl boot-start enable -m systemd -u cribl
sudo systemctl restart cribl-outpost.service

At that point the Outpost is ready for use. A Worker Node connects with only the Outpost listener URL and the Leader auth token, configured from Settings under Global > Distributed Settings > Leader Settings, or via the CRIBL_DIST_LEADER_URL environment variable (copied from the Set Up Cribl Outpost page):

CRIBL_DIST_LEADER_URL=tcp://<leader-token>@<outpost-hostname>:<outpost-port>

Outpost group settings

The Outpost Group Settings page covers connection parameters, retention of disconnected nodes, proxy, storage, and TLS settings. Editing them requires the Admin permission: select an Outpost Group in the sidebar, then go to the Settings tab.

On the Overview tab, Outpost Group Configuration offers a Description, Tags for organizing Outpost Groups into logical categories, and Time to keep disconnected Outposts. It takes formats such as 8h, 5d, or 1w; a bare number defaults to days, and the default is one day.

API Server Settings set where the API server listens: Host, the hostname or IP address it binds to, defaults to 0.0.0.0, and Port, the API port to listen to, defaults to 9000. Each can be overridden with the CRIBL_API_HOST or CRIBL_API_PORT environment variable.

Proxy Settings contain the Use proxy env vars toggle, which honors the HTTP_PROXY/HTTPS_PROXY environment variables and defaults to toggled on. Cribl prioritizes proxy environment variables in this order: Process, User, and System.

The Limits tab provides controls for metrics, storage, metadata, jobs, the Redis cache and connections to it, and CPU settings. Storage offers Min free disk space, the minimum disk space on the host before various features take measures to prevent disk usage, in KB or MB, defaulting to 5 GB. The page points to the "Unable to write to the filesystem, disk space low" error as what to avoid.

Metadata offers Event metadata sources, the list of event metadata sources to enable. The available sources are os, cribl, aws, env, and kube, and no sources are enabled by default.

Two settings carry an explicit consult-Support-first warning. Disable SNI-based connection routing affects how the Cribl Edge control plane routes connections, and Feature Flag Overrides make it possible to disable specific new features if they are causing unexpected issues; the page says not to change either without consulting Support first. Support also sets Log file max size (default 5 MB, minimum 1 MB) and Number of log files to keep (default 5, range 1-100), and only new rotations use that limit.

Distributed Settings include Listener Settings, where Address is the hostname or IP address you want to bind, Port is the API port to listen to, and IP allowlist regex is a regex matching IP addresses that are allowed to establish a connection. The TLS Settings follow the docs for any Source or Destination that supports TLS. Under Service Process, Number of connection listener processes sets the Outpost connection listeners to spawn: positive numbers are an absolute number of listeners, negative numbers are relative to the number of CPUs in the system, and the default is -2.

The remaining options mirror their counterparts in Settings > Global, for logging levels and redactions and security.

What the docs flag

• Enterprise and Linux only. Cribl Outpost requires an Enterprise plan or Enterprise license, and it installs on Linux only.
• No shared host with a Worker. Do not run an Outpost Node on the same host as a Stream Worker.
• TLS moved at 4.16. In versions older than 4.16, Outpost TLS information lived in the individual Outpost Node settings; older Outpost Nodes are placed in the default_outpost Group on upgrade to 4.16. Omitting -S 1 leaves the Outpost to Leader connection without TLS, and TLS must be enabled to reach a Leader in Cribl.Cloud.
• No separate Outpost token. The Outpost does not have a separate auth token, so Node connection always uses the Leader auth token.
• Two Support-gated settings. Do not change Disable SNI-based connection routing or Feature Flag Overrides without consulting Support first.
• Proxy env vars outside systemd. If the Cribl service is managed by a service manager other than systemd, such as upstart or init, the Use proxy env vars toggle might not behave as expected.
• Disk space measures. When free disk space reaches the Min free disk space value, various features take measures to prevent disk usage, so keep available space above the threshold.

Frequently asked questions

What is a Cribl Stream Outpost?

A Cribl Stream Outpost relays communication between Worker Nodes and the Leader. Worker Nodes see the Outpost in the same way as a Leader Node. To connect a Node to the Outpost, you only need to provide the Outpost listener URL and the Leader auth token, because the Outpost does not have a separate auth token.

What are the prerequisites for deploying a Cribl Stream Outpost?

Cribl Outpost requires an Enterprise plan or Enterprise license, and it can be installed on Linux only. The docs also say not to run an Outpost Node on the same host as a Stream Worker, and to create an authentication token on the Leader before deploying an Outpost Node.

How do you add a Cribl Stream Outpost Node?

You can add a new Outpost Node using the bootstrap script provided in the UI, or from a binary file set to the mode-outpost installation mode. A new Outpost Node is placed in the default_outpost group by default, and a different group can be specified with the -g parameter.

What do Cribl Stream Outpost Group settings control?

Outpost Group settings control the listener URL that Worker Nodes use to connect, the TLS settings that secure that connection, and the retention time for disconnected Outposts, along with API server, proxy, storage, metadata, and service process controls. Editing an Outpost Group requires the Admin permission.