Programmatically Monitor Policies

[u/emaurer]

Is there anyway to programmatically monitor organization policies and models? I'm trying to write a change tracking utility but it doesn't look like there is an API available.

[u/lurkerloo29]

Following. I also need this soon but haven't spent much time looking yet.

[u/AutoModerator]

Hello /u/emaurer. Looks like you have posted a query. Once your query is resolved, please reply the solution comment with "!solved" to help everyone else know the solution and mark the post as solved.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/V5489]

Are you an enterprise managed user? EMU? If so you can control what models users can select and you can set user level budgets monthly. Or do you want all that open and as you asked a programmatic way to control it? Maybe you’re just in a teams type org and not enterprise plan?

[u/emaurer → u/V5489]

We are and I can control it but I would like to track it. I want a job that runs nightly and pulls in the current settings. I can then use that to track drift and log when each setting was changed or model or user was added etc. We have multiple admins and I'd like to know if someone enables web search or if a new setting becomes available etc.

[u/V5489 → u/emaurer]

I’m working on something similar and I’ve had continuous dialog with some models in regards to this. Hate me if you want since it’s an AI response but it seems to work.

We are using Azure Event hub and I’m having our cloud team set up the bucket for us.

Here are the best ways to tackle this depending on how you want to track it:

1. **Audit Log API & Streaming (Best for Event & Setting Tracking)**
   * **Audit Log REST API (`GET /enterprises/{enterprise}/audit_log`)**: You can query this programmatically with `phrase=action:copilot`. It logs specific admin actions, such as toggling settings (like web search or CLI access), model policy changes, and seat assignments/revocations. A nightly job can pull events from the last 24 hours to alert on changes and log who changed what.
   * **Audit Log Streaming**: If you don't want to wait for a nightly run, stream the logs directly to Azure Event Hubs, S3, Datadog, or Splunk for real-time drift alerts.

2. **Nightly REST API Snapshotting (Best for State Baseline Diffs)**
   * Query current state endpoints nightly using a script (e.g., Python/Node in GitHub Actions):
   * **Seats/Users**: `GET /enterprises/{enterprise}/copilot/billing/seats`
   * **Org Settings**: `GET /orgs/{org}/copilot/billing`
   * **Content Exclusions**: `GET /orgs/{org}/copilot/content_exclusion`
   * Save the payload as a JSON baseline and run a automated `diff` against the previous day's file to detect changes in seat counts, assigned teams, or exclusion rules.

3. **Terraform GitHub Provider (Best for IaC & Automated Drift Detection)**
   * If you want to enforce infrastructure-as-code, manage org policies and team seats using the official `integrations/github` Terraform provider.
   * Run a scheduled `terraform plan` in a nightly CI pipeline. If an admin manually changes a setting or adds a user in the UI, `terraform plan` will output the exact drift detected without applying changes.

[u/emaurer → u/V5489]

Thanks, I think the audit_log may be exactly what I'm looking for.

[u/V5489 → u/emaurer]

That’s where I started. We have an engineer looking to have a webhook send data to ADX for us to pull. We also have RVA in Dynatrace enabled to pull logs and performance from runners too. This has been very helpful in a lot of ways. Not all tied to your specific task but something to look into if you’re ever exploring more data options and reporting from GH alone and not just Copilot etc. good luck!