OpenAI Compliance Logs with Cribl Stream

Cribl Stream ships a dedicated source for OpenAI compliance data: the OpenAI Compliance Logs Source. This post follows the OpenAI Compliance Logs Source page in the Cribl Stream docs, covering what the source collects, the setup fields and scheduling it exposes, the event types you can request, and the caveats the documentation flags for operators.

What the source collects

Per the docs, the source collects enterprise observability and compliance data from OpenAI's Compliance Logs API. It is intended for organizations using ChatGPT Enterprise that need to feed ChatGPT workspace activity into data loss prevention (DLP) or SIEM tools, as well as to process that data in Cribl Stream before delivery to downstream systems.

The page describes the source as a pull-type source with TLS support and event breaker support. TLS is enabled via HTTPS on the underlying OpenAI Compliance Logs REST APIs.

When a collection job runs, the source performs four documented steps:

1. It calls the Compliance Logs export API with your configured authentication headers and event types selection.
2. It receives a list of files (or equivalent export descriptors) that represent batches of compliance data for the requested types and time window.
3. It downloads and ingests those files so each record becomes an event in Cribl Stream.
4. It tracks state so repeated polls avoid large gaps or unnecessary duplication, consistent with other REST Collectors.

After collection, you can route, filter, enrich, and format events for your downstream Destinations, such as SIEM, observability, storage, or ticketing tools. The docs also list internal fields Stream attaches that you can read in Functions: __collectible carries metadata about the collection job, and __collectStats carries per-request statistics.

Setting up the source

Before configuring the source, the page lists four prerequisites: a Cribl Stream deployment (Cloud or self-hosted); ChatGPT Enterprise (or equivalent OpenAI workspace offering) with access to the Compliance Logs export APIs; a compliance API token (or other credential) that OpenAI expects in request headers, with permission to list and download compliance log exports for your organization; and network connectivity from Cribl Workers to OpenAI's compliance API hosts over HTTPS, direct or via an HTTP/S proxy.

To configure, select Products on the top bar, then Cribl Stream. Under Worker Groups, select a Worker Group, then either navigate to Routing and open QuickConnect to add the source, or select Data and Sources, choose the source, and select Add Source.

The General Settings in the Source modal include the fields below, all as documented on the page:

• Input ID: a unique name for the source. If you clone it, Cribl Stream appends -CLONE to the original Input ID.
• Description: optional, for example which event types or downstream SIEM this instance feeds.
• API key: select or create a stored text secret.
• Account type: whether the source collects compliance logs for a single ChatGPT workspace or for your entire organization. The default is Workspace.
• Workspace ID: if Account type is Workspace, the ID of the ChatGPT workspace to collect logs from, in UUID format.
• Organization ID: if Account type is Organization, the ID of the ChatGPT organization to collect logs from. The page gives org-XXXXXXXXXXXXXXXXXXXXXXXX as an example format.
• Event types: one or more compliance log categories to collect.
• Cron schedule: collection jobs run on this schedule in the Leader's time zone.
• Tags: optional UI tags to group the source in Stream. Tags are not added to events.

The earliest and latest time fields for the schedule use relative offsets. The scheduling section of the Cribl Stream docs page gives the syntax as:

[+|-]<time_integer><time_unit>@<snap-to_time_unit>

Values without units are interpreted as seconds, and the @ unit rounds down from the evaluated time: @d means the start of the current day, while +128m@h moves forward 128 minutes and then snaps back to the hour boundary. The Earliest value must not be later than Latest. The Job timeout accepts values such as 30, 45s, or 15m, and 0 means no timeout; the Scheduling section notes that minimum granularity is often 10 seconds and the default is 300.

Under Event types, a multi-select dropdown chooses which event_type values to pass to the export API, and only the selected types are requested in each collection run. The page's event types table lists these supported values:

• APP_LOG: application-level activity in the ChatGPT workspace.
• APP_AUTH_LOG: application authentication events.
• AUDIT_LOG: administrative and audit trail events.
• AUTH_LOG: user or session authentication events.
• CHATGPT_PLUGIN_SPREADSHEET: plugin activity in the ChatGPT workspace.
• CODEX_LOG: Codex-related usage and events.
• CODEX_SECURITY_LOG: security-focused Codex events.
• CONVERSATION_MESSAGE: conversation message-level records, which the page describes as often the richest content for review workflows.

Scheduling and state tracking for the Collector job are configured with defaults set for typical compliance polling. Retry settings cover failed HTTP requests: retry type defaults to Backoff (with Static or Disabled as alternatives), the initial retry interval has a maximum of 20,000 ms, the retry limit defaults to 5 with a maximum of 20, and the backoff multiplier defaults to 2. Retry HTTP codes typically include 429, 500, and 503, and honoring the Retry-After header is enabled by default. You can also optionally retry on ETIMEDOUT or ECONNRESET for more resilient long downloads.

Advanced settings include the request timeout in seconds (default 300, with 0 meaning wait indefinitely), a time to live for how long Collector job artifacts remain on disk and in Job Inspector (often 4h by default), and a page limit for the maximum pages a single job requests, where 0 sets no explicit page cap. For GitOps, the Environment option can limit the config to a single Git branch. To finish, choose Send to Routes and/or QuickConnect under Connected Destinations, then click Save and Commit & Deploy.

What the docs flag

The first caveat is scoping by account type: when Account type is set to Organization, only APP_LOG and APP_AUTH_LOG are available. The remaining event types are available for Workspace accounts only. The page also advises selecting the minimum set of types that satisfies your DLP or SIEM policy, since requesting every type increases volume and API work.

Compliance exports can produce many files or large payloads. If OpenAI returns rate-limit responses such as HTTP 429, the page's guidance is to rely on the retry settings and a less aggressive schedule, and to reduce the number of event types or narrow the time window per job.

State tracking is how the source prevents overlaps between jobs and gaps in data. The state update and state merge expressions are JavaScript expressions, and the state tracking section of the Cribl Stream docs page breaks down the default state update expression:

__collectible != null && (__collectible.end_time && {lastEndTime: __collectible.end_time > (state.lastEndTime || '') ? __collectible.end_time : state.lastEndTime})

The default values track state by the latest lastEndTime field found in events gathered in a collection run. State values must resolve to an object, shown in the page as {"lastEndTime": "2024-04-05T12:00:00Z"}, and if the expression does not resolve to an object, Cribl Stream ignores the result. You can select Manage State to view, modify, or delete a state.

On errors, non-2xx HTTP responses from the configured endpoints are generally treated as errors. If authentication fails, the page says to verify the token, header names, and any organization or workspace identifiers against OpenAI's current compliance logs documentation. For troubleshooting live, use Live Data and Start Capture on the Source modal to preview events as they are ingested, use the Logs tab on the job or source for request errors and auth failures, and use the Monitoring page to correlate drops in events or bytes with schedule or API errors.

Frequently asked questions

What does the OpenAI Compliance Logs Source in Cribl Stream collect?

It pulls enterprise observability and compliance data from OpenAI's Compliance Logs API. It is intended for organizations using ChatGPT Enterprise that feed ChatGPT workspace activity into DLP or SIEM tools. Stream can then route, filter, enrich, and format the events before delivery to downstream destinations.

Which event types can a Cribl Stream OpenAI Compliance Logs Source request?

The page's table lists APP_LOG, APP_AUTH_LOG, AUDIT_LOG, AUTH_LOG, CHATGPT_PLUGIN_SPREADSHEET, CODEX_LOG, CODEX_SECURITY_LOG, and CONVERSATION_MESSAGE as supported values. Only the types you select in the multi-select dropdown are requested in each collection run.

Why does the OpenAI Compliance Logs Source in Cribl Stream show fewer event types for an Organization account?

When the Account type is set to Organization, only APP_LOG and APP_AUTH_LOG are available. The remaining event types are available for Workspace accounts only.

How does Cribl Stream avoid overlaps and gaps between repeated polls of the OpenAI Compliance Logs API?

The source tracks state so repeated polls avoid large gaps or unnecessary duplication, consistent with other REST Collectors. The default state expressions track the latest lastEndTime field found in events, and a state expression that does not resolve to an object is ignored by Cribl Stream.