SOC 2 Evidence: Log Retention and Replay

Why retention windows are a SOC 2 problem

SOC 2 assessments, built on the AICPA Trust Services Criteria, demand evidence that controls operated over the observation period, and that evidence is mostly log data: who authenticated, who touched configuration, what changed and when. Auditors do not sample the live pipeline. They name a window and expect the production system to produce it: the events for one user, one service, or one incident, across days or weeks.

Cost is the tension: primary indexes are fast, and keeping them hot for long windows is expensive. A common pattern keeps the active window hot, moves everything older to cheap object storage, and pulls back only the slice a request names. That pattern has three parts: a cheap archive for long-period data, a file layout that targets a slice without scanning everything, and a repeatable path back through the same pipeline into the same destinations.

The sections below cover that pattern from two Stream documentation pages, one for the write side and one for the read side.

Sending events to S3

The write side is the S3 Destination documentation. S3 is a non-streaming Destination type: Cribl Stream does not have to run on AWS to deliver data to S3, and S3-compatible stores work with the same type. The Destination can also write to buckets with S3 Object Lock enabled, which matters for an archive kept untouched during assessment.

Two fields split fixed identity from per-event layout. S3 bucket name takes a constant or a JavaScript expression evaluated only at init time, where event-level variables are not available. Put event-level variables in the Partitioning Expression instead; it is evaluated for each file. Partitioning is date-based by default, and a blank expression falls back to the event's __partition field (if present) or to the root of the Output Location and Staging Location.

Data format defaults to JSON (Raw and Parquet are also available), and Compression defaults to gzip, unavailable when the format is Parquet. To prevent files from being overwritten, Cribl appends a random sequence of 6 characters to the end of file names, prefix and suffix expressions included.

Files buffer locally in the Staging location before compression and the move to the final destination. Advanced Settings close a file when any of four limits is met: file size (default 32 MB), open time (300 seconds), idle time (30 seconds), or open file count (100). A staging file limit (default 100, bounds 10 to 4200) caps the files waiting for upload before backpressure, and Backpressure behavior defaults to Block.

The cold side is a settings matter. Storage class defaults to Standard and offers, among others, Intelligent Tiering, Glacier Flexible Retrieval, Glacier Instant Retrieval, and Glacier Deep Archive. Server-side encryption defaults to no encryption, with Amazon S3 Managed Key and AWS KMS Managed Key as the two options. AWS S3 always encrypts data in transit using HTTPS; on other S3-compatible stores, use an https:// URL to do the same.

Authentication method defaults to Auto, which uses the AWS SDK for JavaScript to obtain credentials in a documented order of attempts. Manual accepts a static access key and secret key, which you can store and reference with C.Secret(), and Secret selects a key pair from Cribl Stream's secrets manager. Write permissions are s3:GetBucketLocation when no Region is configured, s3:ListBucket unless Verify if bucket exists is toggled off, and s3:PutObject.

Replaying them back

The read side is the S3 storage and replay guide, which documents Cribl Stream's Replay options: selectively ingest, and re-ingest, data into systems of analysis. Point a Collector at the object store, narrow to the files that matter, and the matching events come back through the system.

What you wrote is what Replay parses. JSON wraps the parsed event, with all metadata and modifications it contains at the time it reaches the Destination step, in a JSON object, one event per line (NDJSON). Raw writes the event's _raw field out as plaintext, unparsed. The guide recommends the default JSON format, and reads it back with the Cribl Event Breaker Ruleset, which contains a newline-delimited-JSON definition.

The path is the filter. Object retrieval and unpacking impose a big resource hit in the Replay process, so minimize the impact radius: path segments map back to variables, including time, so you can zero in on the exact logs you need to replay without requiring checking _raw.

The guide's example Destination builds its path from time and metadata for replay. The Partitioning expression, from the S3 storage and replay guide:

`${C.Time.strftime(_time ? _time : Date.now() / 1000, '%Y/%m/%d')}/${index ? index : 'no_index'}/${host ? host : 'no_host'}/${sourcetype ? sourcetype : 'no_sourcetype'}`

The matching Filename expression, from the S3 storage and replay guide:

`${C.Time.strftime(_time ? _time : Date.now() / 1000, '%H%M')}`

On the write side, the guide wires a Route called Archival matching everything it archives; its example filter is !__replayed, so any event not coming from the defined Replay process gets archived. The Route selects the passthru Pipeline, points at the S3 Destination, and leaves Final unset, so data flows through the Route rather than stopping at it. Position it at or near the top of the Routing table, then verify a green checkmark for it on Monitoring's Data > Destinations.

One constraint shapes the tier choice: the guide states that Collectors and Replay do not support S3 Glacier, S3 Deep Glacier, or the Azure archive tier, due to long retrieval times for those storage classes, and do support S3 Glacier Instant Retrieval when you use the S3 Intelligent-Tiering storage class. Choose the tier with that constraint in mind.

The reader is an S3 Collector: Data > Sources > Collectors > S3 in the top nav (the guide's example ID is Replay), using Auto-populate from to pull the configs from your S3 Destination. The Path field then gets the tokens to extract from the store paths; the S3 storage and replay guide specifies them for the recommended scheme:

/${MYENV}/${_time:%Y}/${_time:%m}/${_time:%d}/${index}/${host}/${sourcetype}/${_time:%H}${_time:%M}-${extended}

That scheme must match the Destination's partitioning scheme. The tokens are what let Cribl Stream immediately exclude, without downloading, files that have no chance of matching the target events, then download and interrogate the contents of what is left and send the matches to the Routing table.

Under Result Settings > Fields, the guide has you create a field named __replayed set to true. As a double-underscore field, it is internal-only and will not be passed on to your final destinations. That closes the loop: the !__replayed filter in the Archival Route keeps replayed events from being re-archived.

The pages drift on one label: the replay guide calls the control the Max open files option, while the Destination page names the same Advanced Settings field the Open file limit.

The guide's test flow starts a capture in Pipelines filtered on __replayed, for 300 seconds, 10 max events, then runs the Collector filtering on true, Earliest -1h, Latest now. In a full run, events proceed through your Routes as normal and land wherever your original Routes and Pipelines dictate.

What the docs do not cover

Neither page says anything about SOC 2, the Trust Services Criteria, or what an auditor accepts as evidence. The source pages do not document a recommended retention period, a mapping from control criteria to the archive or the replay, or a workflow for producing evidence from a stored window.

They also do not document Cribl Stream-side retention or immutability controls. The Destination page states the Destination can write to buckets with S3 Object Lock enabled, but Object Lock is a property of the bucket, and neither page documents a Cribl Stream mechanism to enforce or verify it. The replay guide notes the bucket should grow with your long-term archival needs, but the pages document no expiration, lifecycle, or cleanup behavior on stored objects.

Frequently asked questions

Where does the Cribl Stream S3 Destination keep events before they reach the bucket?

Events are buffered in the Staging location, a filesystem location where files are staged locally before compression and move to the final destination. Files close when any of four Advanced Settings limits is met: file size, open time, idle time, or open file count.

How does Cribl Stream prevent archived files from overwriting each other in S3?

To prevent files from being overwritten, Cribl appends a random sequence of 6 characters to the end of file names. This also applies to prefix and suffix expressions in file names.

What lets the Cribl Stream S3 Collector skip downloading most of an archive?

The Collector's Path field names the tokens to extract from object paths, and the scheme must match the partitioning the Destination used. With them set, Cribl Stream can exclude files that have no chance of matching without downloading them, and only then downloads and interrogates what is left.

Which S3 storage classes can Cribl Stream Replay read back from?

Cribl Stream's Collectors and Replay feature do not support S3 Glacier, S3 Deep Glacier, or the Azure archive tier, due to long retrieval times for those storage classes. They do support S3 Glacier Instant Retrieval when you're using the S3 Intelligent-Tiering storage class.