Ingesting Files and Journal Logs with Cribl Stream

Q: What happens to rotated log files in the Cribl Stream File Monitor Source?

Rotated files that follow the backlog naming scheme resume where the Source left off. Told to tail foo.log, the Source also scrapes foo.log.[0-9] files, and the hashes of the start point and last-read point locate the resume point across a stop and restart. A rename away from the scheme, such as moving foo.log to bar.log, has no findable resume point.

Q: Which file formats can the Cribl Stream File Monitor Source process?

As of 4.2.x, it processes zip (DEFLATE compression only), gzip, zstd, and tar files, UTF-8 text, and non-identifiable binary content. Binary files are broken into base64-encoded chunks and streamed bypassing Event Breakers, and before 4.2.x the Source could handle only text files.

Q: Is the Cribl Stream Journal Files Source enabled by default?

No, Cribl Stream ships the Journal Files Source as a single instance that is disabled by default. Its search path defaults to /var/log/journal/$MACHINE_ID, it streams the primary system.journal it finds there, and its journal allowlist defaults to system.

Q: Which compression does the Cribl Stream Journal Files Source support?

The parser the Source uses to read journal files handles LZ4 and ZSTD compression. When it encounters compressed fields it cannot unpack, an error message emits in the logs.

The File Monitor Source and the Journal Files Source are the two Cribl Stream sources this post covers, and both read data that is already on disk. Both are listed as System type with TLS Support N/A, and both run in Cribl.Cloud only on customer-managed hybrid Workers, not on Cribl-managed Workers. The first reads log files under a search path, per the File Monitor Source page; the second reads the systemd journal, per the Journal Files Source page.

File Monitor

The File Monitor Source generates events from text files, compressed, archived, and (some) binary log files. As of 4.2.x it processes zip (DEFLATE compression only), gzip, zstd, and tar, plus UTF-8 text and non-identifiable binary formats, and it no longer excludes *.gz files by default; before 4.2.x it could handle only text files. Binary files are broken into base64-encoded chunks and streamed, bypassing Event Breakers.

Discovery builds the file list the Source works from: a procedure that runs at a configurable Polling interval (default 10 seconds), a Filename allowlist (default */log/* and *log), and a comparison of each file's current state with previously stored state. Auto mode discovers files that running processes have open for writing; the page warns against pairing it with other file monitors, and notes that Cribl Edge supports Auto mode only on Linux. Manual mode, the default for new Sources, searches a single Search path (an absolute path, no wildcards, default /var/log) down to a Max depth, where 0 covers only the top level and an empty value recurses without limit.

Optional settings add age filtering: a Minimum age duration (formats like 30s, 15m, 1h) before a file starts ingesting, and a Maximum age duration (formats like 30s, 4h, 3d, 1w) that filters events with timestamps newer than the configured duration; a Check file modification times toggle lets the Source skip files by modification time. Collect from end is on by default: the first encounter with a file reads from its end, files discovered afterward are read from the head, and the page says to disable the setting for stateful Event Breakers such as the CSV event breaker.

Rotation handling is hash-based. The Source does not track state by filename; it hashes the head and tail of a file so it can resume where it left off, and told to tail foo.log it also scrapes the foo.log.[0-9] backlog files. It keeps hashes corresponding to the start point within the file and to the last-read point, so if Cribl Stream is stopped, files are rotated, and Cribl Stream restarts, it finds the resume point in those backlog files. A rename within the same backlog scheme, such as foo.log to foo.log.0, resumes at the right place; a rename away from the scheme, such as foo.log to bar.log, has no findable resume point.

Hash length sets how many header bytes feed the identifying hash and defaults to 256 bytes; the page says not to set it above 16384, because that can cause re-ingestion issues, and unadjusted, files with headers larger than 256 bytes look like one file to the Source. Salt file hash mixes in the source file name for files with identical headers, such as CSV files, but enabling it after startup re-ingests all files. Two behaviors bear on testing: copies of the same file in the watched path are not ingested a second time, and multiple Sources pointed at one file emit duplicates, since Sources run independently.

For testing, do not edit ingested files in a text editor: editors move a temporary file into place where the Source assumes appends. The page's recommended append command, from the File Monitor Source troubleshooting section:

echo "{ \"time\":\"$(date -Is -u)\", \"message\":\"...\" }" >> eg.log


            Journal Files
            The Journal Files Source collects data from journald, the centralized logging service of systemd, which the page calls the centralized location for messages from kernel, boot, syslog, and other services on a systemd-enabled Linux system. It is disabled by default, and the page states it is currently configured to collect data from the user journal, as system is privileged. Its Type row lists System, TLS Support N/A, and Event Breaker Support: No.
            Cribl Stream ships a single, disabled instance with a default configuration. Its Search path defaults to /var/log/journal/$MACHINE_ID in Cribl Stream (in Cribl Edge, $CRIBL_EDGE_FS_ROOT/var/log/journal/$MACHINE_ID); it accepts environment variables, and an unassigned $MACHINE_ID is looked up from /etc/machine-id. The default streams the primary system.journal found there, and a Journal allowlist of wildcards (default system) filters the full path of each discovered journal. Discovery re-runs at a configurable Polling interval that defaults to 10 seconds.
            Journals then pass through Filter Rules: events are generated if all the rules' expressions evaluate to true, or if no rules are specified. The default filter expression is severity <= 4, with the description "Allow events having 'emergency', 'alert', 'critical', 'error', or 'warning' priority". Current boot only is on by default and skips events that are not part of the current boot session. An optional Age duration limit (formats like 30s, 4h, 3d, or 1w) sets a maximum age of journal objects to stream, and per the page the Source also filters events with timestamps newer than the configured duration.
            One compression limit stands out: the parser the Source uses reads LZ4 and ZSTD, not all of the compression options journald supports, and when it encounters compressed fields it can't unpack, an error message emits in the logs. The page's example error, copied from the Journal Files Source page:
            Error reading journald event, dropping offset=31678152 evt="Mar 02 21:29:54 goats-xps15 multipassd: undefined"

            Delivery defaults differ by product: QuickConnect in Cribl Edge, Send to Routes in Cribl Stream.
            Choosing between them
            The first split is where the data lives: the File Monitor page presents its Source as ideal for ongoing log file monitoring under a search path; the Journal Files page is scoped to systemd's centralized journal.
            The documented behaviors that differ:
            
              Event Breaking: the File Monitor Type row lists Event Breaker Support as Yes; the Journal Files row lists No.
              Discovery: Auto or Manual with a filename allowlist, versus a single shipped instance whose journals pass a journal allowlist (default system) and optional filter rules.
              Positioning: first-encounter reading from a file's end, plus hash-based resume across rotation, versus a current-boot-only default and an age duration limit.
              Delivery: the File Monitor page states the Source defaults to QuickConnect; the Journal Files page states Cribl Edge defaults to QuickConnect and Cribl Stream to Send to Routes.
            
            Shared ground: a Polling interval defaulting to 10 seconds, and a suppress toggle for missing search paths. The filter itself carries three names across the pages: Filename allowlist, Journal allowlist, and (in the Journal page's discovery summary) Allowed list. On the pages' own terms, the choice tracks where the data sits: files under a directory tree, or the journal of a systemd host.
            What the docs flag
            File Monitor.
            
              No manual editing of ingested files: editors move a temporary file into place, while the Source's logic assumes logs are appended to the existing file.
              Overlapping monitor groups deliver duplicates, which the page says can "run the meter" twice on affected files.
              Chunked transfers such as rsync can be picked up before the transfer completes; the page's fix is serial transfer, for example rsync --append.
              Delete files (Advanced Settings, Manual mode only) removes a source file after the Idle timeout (default 300 seconds), and documents four failure conditions: permissions, files in use on Windows, files moved or renamed while collected, and filenames outside the allowlist.
            
            Journal Files.
            
              Disabled by default: the shipped single instance must be enabled before it collects.
              LZ4 and ZSTD only; other journald compression options emit errors in the logs.
              The page's intro describes collection from the user journal because the system journal is privileged, while its configuration section defaults to the primary system.journal; both statements appear on the page.
            
            Frequently asked questions
            What happens to rotated log files in the Cribl Stream File Monitor Source?
            Rotated files that follow the backlog naming scheme resume where the Source left off. Told to tail foo.log, the Source also scrapes foo.log.[0-9] files, and the hashes of the start point and last-read point locate the resume point across a stop and restart. A rename away from the scheme, such as moving foo.log to bar.log, has no findable resume point.
            Which file formats can the Cribl Stream File Monitor Source process?
            As of 4.2.x, it processes zip (DEFLATE compression only), gzip, zstd, and tar files, UTF-8 text, and non-identifiable binary content. Binary files are broken into base64-encoded chunks and streamed bypassing Event Breakers, and before 4.2.x the Source could handle only text files.
            Is the Cribl Stream Journal Files Source enabled by default?
            No, Cribl Stream ships the Journal Files Source as a single instance that is disabled by default. Its search path defaults to /var/log/journal/$MACHINE_ID, it streams the primary system.journal it finds there, and its journal allowlist defaults to system.
            Which compression does the Cribl Stream Journal Files Source support?
            The parser the Source uses to read journal files handles LZ4 and ZSTD compression. When it encounters compressed fields it cannot unpack, an error message emits in the logs.
            
                Source
                Verified against Cribl Stream 4.20 documentation on May 02, 2026.