Resolving resource group deployment quota limits

Last week we received an odd message when deploying a pipeline about hitting our “deployment quota of 800”. This pipeline has been running every 2 hours for about two months now, roughly 84 deployments a week, it makes sense we would hit the limits regularly now:

“Creating the deployment ‘azuredeploy-20200418-111627-89b8’ would exceed the quota of ‘800’. The current deployment count is ‘800’, please delete some deployments before creating a new one.“

800 deployments to a resource group is quite a lot, Additionally but this is the second time we’ve seen this message in the last 3 months. It’s time to add some code to our pipeline to clean up as we deploy. To resolve, we need to run the following PowerShell to remove deployment history from the Resource Group, in this case, deleting everything older than 30 days.

Write-Host "Get a list of all deployments older than 30 days"
$deployments = Get-AzResourceGroupDeployment -ResourceGroupName SamLearnsAzureQA | Where-Object Timestamp -lt ((Get-Date).AddDays(-30))

Write-Host "removing deployments $deployments.Length"
foreach ($deployment in $deployments) {
  Remove-AzResourceGroupDeployment -ResourceGroupName SamLearnsAzureQA -Name $deployment.DeploymentName
}

This is a very slow command, it takes roughly 60 seconds per deployment history item, about 50-60 items an hour, but it’s important to run this to clean up the backlog before we add it to our pipeline. Once cleaned up, we can add this script to our pipeline. We create a new PowerShell file in our project, with parameters for the resource group. Next we update the YAML, adding a step to run this PowerShell just before we deploy the ARM template.

While the script is slow (a few seconds for each deployment), if we are running it every time a pipeline runs, there shouldn’t be more than a deployment or two per run to clean up.

Finally we commit and push the code and watch the Pull Request complete successfully. Our pipeline is now more robust!

Update: as of June 2020, this has been resolved: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-history-deletions?tabs=azure-powershell

References

• Deployment quota article: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-quota-exceeded
• Featured image credit: https://www.algotech.solutions/blog/wp-content/uploads/2016/02/water-overflow-1440×960.jpg