Posted by Today, we are going to split our automated testing workload across multiple build agents, in both GitHub and Azure DevOps. This is a similar problem we’ve solved in the past, but here we are looking at breaking up our test workloads. Our automated testing duration has crept up over time, now running for about 15 minutes.
Contained within this block of 15 minutes are nearly 100 individual tests. You can see how we broke these down in a previous post about testing strategy. How can we spread the load over multiple agents? Let’s break our test portfolio down first:
- Our web service has 27 unit tests that run in less than 1s
- Our web service has 46 integration tests that run in ~10.5 mins
- Our website has 8 unit tests that run in less than 1s
- Our website has 9 integration tests that run in ~4 mins
MSTest has an attribute you can add to tests, “TestCategory”, where you can categorize different types of tests. We typically use it to test various elements of our test suite, such as unit tests, integration tests, Redis tests, etc. We are going to use that category to split our integration tests into two batches. Note: These features are available in any .NET Core or .NET Framework version, as well as variations of MSTest, nUnit, and xUnit. The syntax in a MSTest test looks like this:
[TestCategory("UnitTest")]
If you are using multiple test categories, the syntax is:
[TestCategory("UnitTest1"), TestCategory("UnitTest2")]
In our yaml we can then add a line to the dotnet tests arguments to filter by TestCategory:
--filter TestCategory=UnitTest
This syntax also supports “or” and “and” functionality:
- or: –filter TestCategory=UnitTestA|TestCategory=UnitTestB
- and: –filter TestCategory=UnitTestA&TestCategory=UnitTestB
Using these test categories, we have spread our tests across 4 jobs (1 for unit tests, 2 for the service integration tests, and 1 for website integration tests). These changes make a significant difference to overall time, the build now runs in less than 5 minutes (a saving of 10 minutes)!
How does this work on GitHub?
Of course, this works This is definitely true in GitHub too. In our feature flags project, automated tests already run in a few minutes, but we can still save time by splitting our tests across multiple jobs. Before we make these changes, the entire pipeline runs in 20 minutes, with 10 and a half minutes for building and testing:
After moving the tests to a separate job, the total time drops to just under 14 minutes, success!!
Wrap-up
Parallel jobs continue to be useful in solving many situations where we just have large jobs that need to be spread out – “many hands make light work!” Particularly as many teams continue to struggle to reduce over all test time, parallel jobs helps to keep our overall DevOps pipeline performance optimized, improve developer productivity, and reduce that feedback loop.
References
- dotnet test with testcategory: https://docs.microsoft.com/en-us/dotnet/core/testing/selective-unit-tests?pivots=mstest
- Multiple test categorys: https://docs.microsoft.com/en-us/previous-versions/dd286683(v=vs.140)?redirectedfrom=MSDN
- dotnet test filter docs: https://docs.microsoft.com/en-us/dotnet/core/testing/selective-unit-tests?pivots=mstest
- Featured image credit: https://bitbar.com/wp-content/uploads/2019/12/parallel-real-device-and-browser-testing.png
Sam Learns Azure 
One comment