Audit streaming with Event Grid

Posted by

In Azure DevOps, there is an audit log to record changes to a variety of events in your Azure DevOps instance. By default, this data is retained for 90 days. but what if we need to retain this data for longer? Some organizations need to maintain this data for years. Today we are going to export the logs with “audit streaming” and retain this data forever, using an Event Grid to process the logs, with a Azure Function, and then store the data into a Azure Storage table.

Azure DevOps Auditing

To complete this addition, we will need to be a Azure DevOps organization administrator. Let’s start by browsing to the “auditing” section in our organization. Here we can see the most recent events.

An individual event contains information about the activity, user, location, etc. An event from our pipeline is shown below, where it logged myself accessing the audit log.

To setup an automatic feed to retain these in an Azure storage table for a longer term, we need to browse to the streams tab, and click on the “New stream” button.

After clicking on the new stream button, we can see three options.

  • Azure Event Grid is the easier option for us to implement, in our opinion. It has the most control and least number of restrictions.
  • Azure Monitor Logs requires Azure Sentinel, which is extra infrastructure we are not willing to deploy
  • Splunk: We don’t use Splunk, so that isn’t an option

To setup to an Event Grid, we will need to create one first.

Creating Event Grid

In Azure, we start by adding a new “Event Grid Topic” resource.

On the first page, we create a new resource group, add a Event Grid name, and select the resource location.

We then select the “Event Grid Schema”, and “enable system assigned identity”, to setup a managed identity.

Next we need to create a storage account.

With the Event Grid and Azure Storage created, we browse to the Event Grid to check it out. We will need the “topic endpoint” link to configure Event Grid in out Azure DevOps organization, which we copy and save for later.

We will also need an Event Grid access key.

Next we need an Azure function, which we will use with a Event Grid trigger to read in the messages from the Event Grid and push them to an Azure Storage table. The code for this is just a few lines, and can be viewed here.

Now we create the Event subscription. After we name it, we select an Azure function as a endpoint type, and the function we just created.

Final Azure DevOps configuration

Now back in Azure DevOps, we can finally select the Event Grid option, and paste in the endpoint url and access key we saved much earlier.

That is it! We let Azure DevOps log for an hour, and then back in Event Grid we can confirm that the events are being logged successfully.

Wrap-up

Today we setup long term Azure DevOps audit log retention, using an Event Grid, Azure Function, and Azure Storage table.

References

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s