.NET Core + xUnit + Coverlet + ReportGenerator + Cake => Cross-platform Code Coverage Report

7 min readDec 23, 2018

Our Xamarin team has recently delivered a UWP application. During the project set-up phase we spent some time discussing which CI&CD server to use taking into account all the requirements we had back then. Since our goal was a UWP app we decided to go with Azure DevOps, formerly known as Visual Studio Team Services (VSTS), as it seamlessly integrates with other Microsoft technologies.

One of our concerns was unit-tests code coverage calculation. We were not able to go with the straightforward and easy approach and use Analyze Code Coverage feature integrated in Visual Studio and Azure DevOps without having Visual Studio Enterprise subscriptions.

So we decided to find an approach which utilizes open-source projects.

Let me share my findings on this matter and walk you through the steps to build and configure a cross-platform .NET Core solution with several test-projects and an ability to easily generate a code coverage report on your local machine as well as inside an Azure DevOps build pipeline.

Preparation: creating a solution with 2 projects + 2 test projects

You can complete all the following steps either on a Mac or a Windows machine, so it doesn’t really matter here. Let’s start with creating a solution folder and adding a Visual Studio Blank Solution file inside it, then create ‘src’, ‘tests’, and ‘scripts’ folders inside the solution folder.

Let’s add two .NET Standard 2.0 projects (abstract names ‘Common’ and ‘Domain’ were used just as an example) and two corresponding .NET Core xUnit test projects.

Extend the tests projects with references to Coverlet nuget package and to a corresponding project for testing.

CodeCoverageCalculation.Common.Tests.csproj

Writing code and unit tests

Let’s add ‘Utils’ folder to ‘CodeCoverageCalculation.Common’ project and write a simple class containing an extension method.

Let’s move on to writing unit tests for this extension method. These are the first few tests which came to my mind:

Most of articles on the Internet show a basic case when there is only one test project in a solution, but the odds are that in real life you’ll need a more generic approach for code coverage calculation when there are 2+ test projects. For the sake of showing how to calculate code coverage for such a case, I’ve added one more trivial class and one parametrized test for it.

Let’s run the tests to ensure that everything works fine so far.

I believe that we are good on that part and ready to move on and find out how much code we’ve covered with these unit tests.

Writing a Cake script to calculate code coverage

The articles that I’ve seen on the Internet describe how to achieve the same goal with PowerShell scripts, but I decided to try and learn how to use Cake build automation system. There were two reasons for this choice. Firstly, Cake is a cross-platform build automation system meaning that you need to write a script only once and it’ll be run everywhere. Secondly, it enables you to write your build scripts in C# which is pretty convenient. The documentation suggests using Visual Studio Code with Cake Extension, so I was using them.

Install the extension and open ‘scripts’ folder in VS Code. Open ‘Show All Commands’ pop-up by pressing ‘Cmd+Shift+P’ on Mac or ‘Ctrl+Shift+P’ on Windows and choose ‘Cake: Install a bootstrapper’ option.

Select the first option, then repeat the steps to select the second one as well.

I changed the names of the generated files replacing ‘build’ with ‘run_tests_and_generate_coverage_report’. Then I created ‘run_tests_and_generate_coverage_report.cake’ file. As the name of the cake script is not default (the default one is ‘build.cake’), we need to go into the generated ‘*.ps1’ and ‘*sh’ files and replace ‘build.cake’ with the name of our cake file. Now we are ready to write the cake script’s logic. This is the code that I came up with.

The script has two dependencies for passing coverlet arguments in a strongly typed manner to Coverlet and Report Generator. The first dependency is Cake.Coverlet add-in, the second one is ReportGenerator build-in cake tool.

There are a few nuances in the script that I’d like to draw your attention to.

The first one resides inside ‘Test’ task. The responsibility of the task is to generate code coverage output in a way that’ll suit us further down the road. There are a bunch of coverlet output formats available with ‘json’ being the default one. If you use the default output format, the results won’t be understood by Azure DevOps in the opposite to ‘cobertura’ format. So, the goal of the task is clear. We need a single ‘cobertura’ file containing all the coverage results, meaning that the results of the code coverage for each test project have to be merged. The good news is that coverlet supports merging, the bad news is that there is an inconvenience:

With Coverlet you can combine the output of multiple coverage runs into a single result, but the value given to -merge-with must be a path to Coverlet's own json result format.

I’ve worked around this issue by using a temporary ‘json’ output format and ‘merge-with’ option for the first N-1 projects and using ‘cobertura’ output format for the output of the last merge.

Another nuance is inside ‘Report’ task. The task itself is responsible for converting a ‘cobertura’ xml report to a human readable form (HTML in our case). The thing is that, if you are going to use your coverage reports not only locally but also integrate them into an Azure DevOps pipeline, you have to use either ‘HtmlInline’ or ‘HtmlInline_AzurePipelines’ (UI will be a bit different) ‘reportTypes’ value. You wouldn’t want to use ‘Html’ option as Azure DevOps have security constraints on this point and won’t let the stylings work.

Now let’s execute ./run_tests_and_generate_coverage_report.sh in terminal (MacOS case) or.\run_tests_and_generate_coverage_report.ps1 in PowerShell window (Windows case). For PowerShell you may also need (if you haven’t done it yet) to change the user preference for the PowerShell execution policy. It determines which scripts, if any, must be digitally signed before they will run.

As soon as the script has done its work you’ll be able to find the results inside ‘code_coverage’ folder which is created by the script. Open ‘index.htm’ file and observe the statistics, navigate deeper if you’re interested in details.

Integration with Azure DevOps

Our team created a private project on Azure DevOps and set up a self-hosted build&release agent on one of our machines to avoid the build time limitation of the free plan and to speed up our builds.

I created a simple demo project on Azure DevOps which doesn’t have any unnecessary distractions. Let’s take a look at its build pipeline.

We can use Cake Azure DevOps extension to execute our cake script in an easy way as it is demonstrated on the screenshot below.

In addition, we need to use standard Azure DevOps tasks to publish the results of tests’ execution and to publish the code coverage information. Take a look at the tasks on the following screenshots.