Posted by In Azure DevOps, there is an audit log to record changes to a variety of events in your Azure DevOps instance. By default, this data is retained for 90 days. but what if we need to retain this data for longer? Some organizations need to maintain this data for years. Today we are going to export the logs with “audit streaming” and retain this data forever, using an Event Grid to process the logs, with a Azure Function, and then store the data into a Azure Storage table.
Azure DevOps Auditing
To complete this addition, we will need to be a Azure DevOps organization administrator. Let’s start by browsing to the “auditing” section in our organization. Here we can see the most recent events.
An individual event contains information about the activity, user, location, etc. An event from our pipeline is shown below, where it logged myself accessing the audit log.
To setup an automatic feed to retain these in an Azure storage table for a longer term, we need to browse to the streams tab, and click on the “New stream” button.
After clicking on the new stream button, we can see three options.
- Azure Event Grid is the easier option for us to implement, in our opinion. It has the most control and least number of restrictions.
- Azure Monitor Logs requires Azure Sentinel, which is extra infrastructure we are not willing to deploy
- Splunk: We don’t use Splunk, so that isn’t an option
To setup to an Event Grid, we will need to create one first.
Creating Event Grid
In Azure, we start by adding a new “Event Grid Topic” resource.
On the first page, we create a new resource group, add a Event Grid name, and select the resource location.
We then select the “Event Grid Schema”, and “enable system assigned identity”, to setup a managed identity.
Next we need to create a storage account.
With the Event Grid and Azure Storage created, we browse to the Event Grid to check it out. We will need the “topic endpoint” link to configure Event Grid in out Azure DevOps organization, which we copy and save for later.
We will also need an Event Grid access key.
Next we need an Azure function, which we will use with a Event Grid trigger to read in the messages from the Event Grid and push them to an Azure Storage table. The code for this is just a few lines, and can be viewed here.
Now we create the Event subscription. After we name it, we select an Azure function as a endpoint type, and the function we just created.
Final Azure DevOps configuration
Now back in Azure DevOps, we can finally select the Event Grid option, and paste in the endpoint url and access key we saved much earlier.
That is it! We let Azure DevOps log for an hour, and then back in Event Grid we can confirm that the events are being logged successfully.
Wrap-up
Today we setup long term Azure DevOps audit log retention, using an Event Grid, Azure Function, and Azure Storage table.
References
- Audit logs: https://docs.microsoft.com/en-us/azure/devops/organizations/audit/azure-devops-auditing?view=azure-devops&tabs=preview-page
- Audit streaming: https://docs.microsoft.com/en-us/azure/devops/organizations/audit/auditing-streaming?view=azure-devops
- Events tracked: https://docs.microsoft.com/en-us/azure/devops/organizations/audit/azure-devops-auditing?view=azure-devops&tabs=preview-page
- Azure Function code: https://dev.azure.com/samsmithnz/SamLearnsAzure/_git/AzureDevOpsAuditStreaming
- Adding the role assignment for table queue: https://stackoverflow.com/questions/63076867/how-to-use-azure-identity-clientsecretcredential-to-access-a-azure-queue
- Featured image credit: https://www.fairwarning.com/wp-content/uploads/2017/08/who-what-when-and-where-the-power-of-the-audit-trail-in-data-security-fairwarning-blog.jpg
Sam Learns Azure 
One comment