GDPR and Data Minimization in Logging Pipelines

Data minimization, in one paragraph

GDPR's storage limitation principle, Article 5(1)(e), requires personal data to be kept in a form that permits identification of data subjects for no longer than necessary, and Article 25 adds data protection by design and by default: process only what the stated purpose actually needs. In a logging environment, raw events carry SSNs, card numbers, tokens, and other identifiers that no downstream purpose needs, and the pipeline is where those fields can be redacted, sampled, or steered to cheaper, more controlled storage before the data settles.

Mask before it stores

The Mask Function replaces patterns in events, a process the docs call masking, and the Mask Function page says masking is "especially useful for redacting PII (personally identifiable information) and other sensitive data". The Function uses the JavaScript (ECMAScript) regex engine, so when you test patterns in an external tool, the page says to select the ECMAScript (JavaScript) flavor. Each masking rule pairs a Match Regex, which supports capture groups and the /g flag to replace all matches, with a Replace Expression, a JavaScript expression or literal that can reference the capture groups as g1, g2, and so on. The Apply to fields list controls where the rules run: it defaults to _raw, supports wildcards and nested addressing, and matches double-underscore internal fields only when they are enumerated individually rather than through wildcards.

The load-bearing line for a minimization argument: "Cribl does not retain any content that matches the Match Regex that is not inserted into the Replace Expression". Anything a rule matches that the expression does not write back is gone from the event. The page's Social Security number example assumes _raw fields carry the Key=Value pattern social=#########, and its rules, copied from the Mask Function docs:

Match Regex: (social=)(\d+)
Replace Expression: `${g1}${C.Mask.md5(g2)}`

There, social= is capture group g1 and the number is g2, so the output keeps the key and hashes the value. The page warns against folding the key into the match: "Without the key name preceding the hashed value, it isn't clear what value is being hashed". The credit card example has the same shape, its pattern tolerating hyphens and spaces as separators; its rules, copied from the Mask Function page:

Match Regex: (cardNumber=)(\b(?:\d[ -]*?){13,16}\b)
Replace Expression: `${g1}${C.Mask.md5(g2)}`

The Replace Expression is a full JavaScript expression: it can branch on capture groups, reference other event fields with an event. prefix (required in this input alone), and its output is a string even when the expression produces a number, in which case the page points to a Numerify Function to convert the value back. For split audiences, the page says that if you need to send the unmodified values to a destination, such as an archival store, you can use the Destination field for the associated Route to narrow the scope of the Mask Function.

Sample what you must keep

The Sampling Function "filters out events, based on an expression and a sampling rate", and each Worker Process executes the Function independently on its share of events, per the page's note on the shared-nothing architecture. Its Filter defaults to true so it evaluates all events, its Description defaults to empty, and its Final toggle, off by default, stops feeding data to downstream Functions.

Each sampling rule carries its own Filter expression, with true matching all events, and a Sampling Rate: an integer N that defaults to 1, where sampling picks 1 in N of the events that match the rule. The page's worked example sets the rate to 30, meaning "only 1 of every 30 events would be kept", and a capture of 100 events from a datagen Source kept about 4, or "close to 1 in 30". The Better Practices topics the page links frame the use as sampling verbose or voluminous data at ingest time.

Store the rest cold

Masking and sampling decide what stays in the event and how many events stay; the Data Lake Amazon S3 Destination decides where the kept events land. It "sends data to Amazon Simple Storage Service (Amazon S3) and uses a partitioning scheme that is designed to work with Cribl Search", and it is a non-streaming destination with TLS support. Its general settings are an Output ID and an S3 bucket name, which "can be a constant, or a JavaScript expression that will be evaluated only at init time".

Under Partition by fields you name event fields to partition the object path by, in addition to time, which is included automatically. The scheme prefixes objects with alpha characters so that S3's default ascending sort returns them in 'newest-first' order, which per the page lets Search find files that will likely have the requested objects soonest. The effective partition, copied from the Amazon S3 Destination page:

hjhh-2023/ag-07/ai-25/aj-14/<list/of/fields>

The Auto method uses the AWS SDK for JavaScript and tries, in order, environment variables, IAM Identity Center (SSO), the shared credentials file, IAM roles for EC2 and ECS, a JSON file on disk, and other credential-provider classes. Manual takes a static access key and secret key, each a constant or a JavaScript expression that references a stored secret. The page warns that Azure-based Cribl.Cloud Worker Groups do not have AWS credentials, so Auto finds none and authentication fails on those groups.

The storage-class setting defaults to Standard and offers Reduced Redundancy Storage, Standard-Infrequent Access, One Zone-Infrequent Access, Intelligent Tiering, Glacier, and Deep Archive for uploaded objects; server-side encryption for uploaded objects defaults to no encryption, with S3-managed or KMS-managed keys as the alternatives. Files are closed and moved to their final location when any of four conditions is met: the File size limit in MB (defaults to 32), the File open time limit in seconds (defaults to 300), the Idle time limit in seconds (defaults to 30), or the Open file limit (defaults to 100), the last one evaluated on any individual Worker Process.

What the docs do not cover

None of the three pages mentions GDPR, and none describes what happens to stored data after it has been written. The Mask Function page covers processing-time field changes: matching content the Replace Expression does not re-insert does not remain in the event, and Evaluate fields can tag the events a rule matched for downstream processing and reporting, but the page says nothing about expunging already-stored data, handling a data subject access or erasure request, or whether a hashed value still amounts to personal data. The Sampling Function page documents which events are kept and which are dropped in flight; it does not describe recovering dropped events, holding data under a legal obligation, or how a chosen rate interacts with a retention schedule. The S3 Destination page documents how files are staged, closed, and uploaded: the file-close conditions, storage class, encryption, dead-lettering for undeliverable files once retries are exhausted, and cleanup of empty staging directories. Its IAM permission list (s3:ListBucket, s3:GetBucketLocation, s3:PutObject, plus two KMS permissions for multipart uploads) contains no delete right, and the page does not state how to delete objects, expire data, place data on legal hold, or what audit records the destination itself produces.

What the pages establish is the Cribl-side mechanics: what a rule removes, which events a rate keeps, and where kept events land. Whether those mechanics satisfy a given obligation is a controller decision made on top of them, and these pages do not take that call.

Frequently asked questions

For GDPR data minimization, does the Cribl Stream Mask Function remove matched values from events?

Yes: the page states that Cribl does not retain any content that matches the Match Regex that is not inserted into the Replace Expression. In the Social Security number example, the rule keeps the social= key and replaces the number with its md5 hash, so the original digits no longer remain in the event. The replacement value is written out as a string even when the expression produces a number.

If a data subject's records must not be kept, what does the Cribl Stream Sampling Function do?

It filters out events based on an expression and a sampling rate, keeping 1 in N of the events that match each rule. Each Sampling Rate is an integer that defaults to 1, and the page's rate-of-30 example kept about 4 of 100 captured events, which the page calls close to 1 in 30. Which events are filtered out depends on each rule's own Filter expression.

Does the Cribl Stream Amazon S3 destination page describe deleting or expiring objects after upload?

No, it does not describe deleting objects, expiring data, or placing data on legal hold. What it documents is the write side: the four file-close conditions, the storage class of uploaded objects, server-side encryption, and dead-lettering for files that cannot be delivered. Its own IAM permission list contains no delete right.

How does Cribl Stream keep raw values out of the destinations that only need masked data?

It narrows the scope of the Mask Function per route. The page says that if you need to send unmodified values to a destination, such as an archival store, you can use the Destination field for the associated Route to narrow the scope of the Mask Function. Within that scope, each rule still carries its own Match Regex and Replace Expression.