Project Wise Code Coverage Thresholds

Code Coverage and asserting on code coverage thresholds is an important aspect of engineering fundamentals of every project written in any language. Although generally, more the code coverage the better it is but the effort which goes in increasing code coverage beyond a threshold of lets say 90 should be questioned. Some of the questions one needs to ask before going for high code coverage is:

1. What is the value add of having 90 or hundred percent code coverage?
2. Should we invest time in mocking components just to increase code coverage?
3. Code which doesn’t have business logic, should that be unit tested at all?
4. Should we test or include in tests rather sovereign other party components like our databases and messaging components?

If we think about the above 4 questions, we would be inclined towards a very logical conclusion that not all code has business logic, some code may have very less business logic and some code may have high percentage of business logic. But how does this thinking helps us in unit testing?

Lets take an example.. a C# azure devops repository which has several projects in it, for example Sample.Core, Sample.API, Sample.Application, Sample.Repository. Many C# projects are divided in such a structure where each layer of code interacts with the other layer of code (which is famously known as onion architecture- https://social.technet.microsoft.com/wiki/contents/articles/36655.onion-architecture-in-asp-net-core-mvc.aspx). In this type of structure we will notice that not all projects will have same percentage of business logic. Business logic will mostly be present in Sample.Application and projects like Sample.Repository will have very less business logic. Hence it makes sense that we write a lot of unit test cases for Sample.Application and very less unit test cases for Sample.Repository. So we have something like the following:

1. Sample.Application may have 95 percent code which should be unit tested.
2. Sample.API may have 70 percent of the code which should be unit tested
3. Sample.Core may have 50 percent of the code which should be unit tested
4. Sample.Repository may only have 10 percent of the code which should be unit tested.

Now how do we assert these 4 different values in our build pipeline. The popular “Build Quality Checks” azdo task doesn’t let us define code coverage thresholds on project level, hence we have developed our own azdo task https://marketplace.visualstudio.com/items?itemName=LakshayKaushik.CodeCoverageThresholds which lets you define code coverage on project level along with asserting on cumulative code coverage of all projects.

How to use this task?

1. Install the below extension in your azure devops org:

Code Coverage Thresholds - Visual Studio Marketplace

2. Select codecoveragethreshold task from pipeline tasks.

- task: codecoveragethreshold@1
  inputs:
    projectNames: 'project1,project2,project3'
    projectThresholds: '10,15,20'
    codeCoverageFile: 'Cobertura.xml'
    codeCoverageFormat: Cobertura
    coverageCalculation: line-rate

projectNames- Accepts either 'Overall' or a comma seperated list of all projects for which code coverage threshold needs to be defined.

projectThresholds- Accepts a comma seperated list of threshold values for projects. Index 0 of this list corresponds to index 0 of projectNames list.

codeCoverageFile- File which contains the code coverage results. For e.g. Cobertura.xml

codeCoverageFormat- Format of the code coverage report. By default Cobertura is supported.

coverageCalculation- Code coverage calculation mechanism. Both line-rate and branch-rate are supported. Default is line-rate if coverageCalculation is not defined.

Sample Outputs from the codecoveragethreshold task:

Output of task when some Projects fall short of defined threshold:

Output of task when all projects are above code coverage threshold:

Output of task when Overall Code Coverage is above the defined threshold:

Output of task when Overall Code Coverage is below the defined threshold:

Source code of the task:

https://github.com/lakshaykaushik/AzdoCodeCoverageThreshold