Parallel jobs in PowerShell

As part of our post on using parallel jobs in Azure DevOps, we also wanted to focus on some of the individual tasks. One of those was a PowerShell file we run to populate secrets into Azure Key Vault. Wouldn’t it be great if we use multi-threading for our PowerShell? It turns out there is PowerShell construct, known as ‘jobs’, that is similar to multi-threading. It’s not perfect, and in a few situations we found it doesn’t help, but there are many places it does help.

Take this sample. It’s broken into 5 parts. The first and last parts measure the total time of the script. The middle 3 parts, get the time, wait 10 seconds, and then report out the time afterwards.

Write-Host "Starting script $(get-date)"

$startJobTime = get-date
sleep 10
Write-Host "Job 1 $startJobTime to $(get-date)"

$startJobTime2 = get-date
sleep 10
Write-Host "Job 2 $startJobTime2 to $(get-date)"

$startJobTime3 = get-date
sleep 10
Write-Host "Job 3 $startJobTime3 to $(get-date)"

Write-Host "Ending script $(get-date)"

All in all, this runs in serial, in about 30 seconds.

What if we could wrap up each of these 10 second jobs into it’s own thread? Meet PowerShell jobs. With a job, we can wrap up a script and start a script (with “start-job”), and then continue executing our script. Later in our script, we use “Get-Job | Wait-Job” to get all jobs – after waiting for our job(s) to complete. At the end of the script we have some scripts to show the output of each job, and then clean up and remove the job.

$startScriptTime = get-date
Write-Host "Starting script $startScriptTime"

#Start each script as a job
$job1 = start-job -ScriptBlock {
$start = get-date
sleep 10
$end = get-date
Write-Host "Job 1 $start to $end"
}

$job2 = start-job -ScriptBlock {
$start = get-date
sleep 10
$end = get-date
Write-Host "Job 2 $start to $end"
}

$job3 = start-job -ScriptBlock {
$start = get-date
sleep 10
$end = get-date
Write-Host "Job 3 $start to $end"
}

#Wait for jobs to finish
Get-Job | Wait-Job

#Output results
Receive-Job $job1
Receive-Job $job2
Receive-Job $job3

#Clean up jobs
remove-job $job1.id
remove-job $job2.id
remove-job $job3.id

$endScriptTime = get-date
Write-Host "Ending script $endScriptTime"

How long does this script take? It doesn’t quite run in 10 seconds, but it’s very close, finishing in ~11 seconds, still a significant upgrade from our 30 seconds before

What is the downside?

As we mentioned earlier, results can be mixed. In our project, we have a PowerShell script to populate our Azure Key Vault with secrets that runs in about 20 seconds. We use this after we deploy our ARM Templates, to upload the output variables generated. The script calls Azure Key Vault 6 times, (twice to set access policies, and four times to set actual secrets).

• Access Policies: ~6 seconds per call (10-12 seconds total)
• Set Secrets – ~2 seconds per call (8 seconds total)

At first glance, it looks like we might be able to drop the time down to 6-7 seconds, but with jobs, the time increased to ~25 seconds. We tried a few different combinations, but ultimately had to accept that 20 seconds without jobs is simpler and faster than 25 seconds with jobs.

Wrap-Up

Another tool in our toolbox next time we have a long running PowerShell file!

References

• Microsoft start-job docs: https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/start-job?view=powershell-7
• Featured image credit: https://cdn-az.allevents.in/events1/banners/707031adab9d1dc88f0268431f461720b89c8472a1aaabe9f1da28e349254fad-rimg-w940-h540-gmir.jpg?v=1566555285