Code Coverage and Web Tests

Download sample code for this blog entry here.

Code Coverage can be a helpful metric to see how much of your code is hit by the tests you have written.  Although it isn't a way to measure how good your code is, it will at least help you to see what parts of your code have no automated tests written against them.

However, you can run into some difficulties when trying to do code coverage for tests that hit a web site.  Let's look at an example.  Suppose you are creating a web application, and you want to write tests against it.  Some of the tests will be unit tests that test out specific classes in isolation.  Other tests will be integration tests that use HTTP to test the website.  You can see the sample solution structure at the right.

First, we're going to test the SampleLogic class in the Logic project.  Here is the SampleLogic class:

    public class SampleLogic
    {
        public int val;
 
        public SampleLogic(int value)
        {
            this.val = value;
        }
 
        public int Double()
        {
            return val * 2;
        }
    }

Nothing too complex here - we're doubling a number.  So let's set up code coverage.  Under Solution Items, double-click on 'Local.testsettings', then under 'Data and Diagnostics' enable Code Coverage.

Then double-click on the Code Coverage line, and choose to instrument the CodeCoverageDemo and the Logic assemblies.  This will allow us to see what percentage of the code in these assemblies is touched in some way by the tests that we run.

Click OK, then Apply, Then Close.  Code Coverage is now enabled for the web project.  Next, open up the UnitTests.cs file in the Tests project.  Here is what it the UnitTests class looks like:

    [TestClass]
    public class UnitTests
    {
        [TestMethod]
        public void SampleUnitTest()
        {
            var sample = new SampleLogic(4);
            var result = sample.Double();
            Assert.AreEqual(result, 8);
        }
    }

Right-click on the SampleUnitTest method, then click Run Tests.  This should run only the Unit Tests.  Based on the code seen above, it would make sense to assume that we'll have 100% code coverage for the SampleLogic class, and 0% coverage for the CodeCoverageDemo web project assembly.

And sure enough, that's what we get (the CodeCoverageDemo assembly doesn't show up because it's 0% covered).  The SampleLogic class has 100% code coverage.  So far, so good.  Now let's look at the integration test. We'll be testing the loading of the Demo.aspx page.  It's important to see the Page_Load method in the codebehind for Demo.aspx:

        protected void Page_Load(object sender, EventArgs e)
        {
            var sample = new SampleLogic(5);
            var sampleDoubled = sample.Double();
 
            Label1.Text = "Doubled Value is " + sampleDoubled.ToString();
        }

This method should touch every line of code in the SampleLogic class.  Now let's look at the integration test in the WebTests.cs file:

        [TestMethod]
        public void SampleWebTest()
        {
            var client = new WebClient();
            var result = client.DownloadString("http://localhost:7654/Demo.aspx");
 
            var contains = result.Contains("Doubled Value is 10");
            Assert.AreEqual(contains, true);
        }

Again, I would expect to see 100% code coverage after running this test.  Be sure to start the web site, then run the integration test.

This is where things get a little strange.  If I'm running the web site in Cassini or IIS Express, it works great and I see 100% code coverage for both assemblies, as seen here:

However, if the site is running in IIS, Visual Studio doesn't seem to be able to pick up on the code coverage, as seen here:

So what do we do about this?  There are several options:

1. Use a third party tool to help get code coverage metrics.
2. Ensure that you have both unit and integration tests for your classes.  The integration tests are important to ensure that the web methods and pages are working from end to end, and the unit tests are important to test each piece independently.  The unit tests will be reflected in your code coverage metrics.

3. Use IIS Express or Cassini instead of IIS


Code Coverage with Distributed Builds and Web Tests
If you use TFS for your builds, and TFS is set up so that the web server is on a different box from where the tests are running, then the machine running the tests will have zero ability to determine code coverage for the DLLs on the web server, and you may see this error:

           Cannot open the ASP.Net project 'MyProjectName'

If you see this error, you can either reconfigure TFS so that the test runner has the website running locally, or you can change the Code Coverage configuration so that it does not attempt to measure Code Coverage for the web projects.

From http://msdn.microsoft.com/en-us/library/dd286743.aspx:

You can use code coverage only when you run an automated test from Microsoft Visual Studio 2010
or mstest.exe, and only from the machine that runs the test.  Remote collection is not supported.

Bonus Tip:
You can either run with debugging or run with code coverage - you can't do both at the same time.  If you try to do both, you'll see a message like this:

"Debugging tests running on a remote computer or with code coverage enabled is not supported.  
The tests will run under the debugger locally and without code coverage enabled."

Other Helpful Resources:
https://connect.microsoft.com/VisualStudio/feedback/details/650894/running-unit-tests-with-code-coverage-in-vs-2010-sp1-with-iis-express-hosted-project#tabs
http://blogs.msdn.com/b/colinth/archive/2010/07/23/tip-fixing-vsperfaspnetcmd-metabase-errors.aspx
http://weblogs.asp.net/gunnarpeipman/archive/2010/05/04/how-to-configure-visual-studio-2010-code-coverage-for-asp-net-mvc-unit-tests.aspx