This is one in a series of articles about Time Tortoise, a Universal Windows Platform app for planning and tracking your work schedule. For more on the development of this app and the ideas behind it, see my Time Tortoise category page.
I found an interesting technique this week related to the NuGet/xUnit.net dependency issue that I have written about previously. It doesn’t solve the dependency problem, but it’s another tool that can be used when Visual Studio or xUnit fail to discover unit tests.
Review
Here’s a summary of the problem: When using xUnit.net to test a .NET Core or .NET Standard class library that depends on NuGet packages, the DLLs associated with those packages aren’t automatically copied to the test class output directory. This can result in tests that are not discovered (i.e., they don’t show up in Visual Studio Test Explorer), or that are discovered but fail to run (due to missing assemblies). Manually copying the DLLs fixes the problem, but finding and copying DLLs is supposed to be NuGet’s job.
As I was searching for a potential solution to this problem, I came across Getting started with xUnit.net (.NET Core / ASP.NET Core), the official xUnit.net documentation on testing .NET Core assemblies. It contains steps that are relevant to the dependency issue. After some experimentation, I tried the following on the Time Tortoise solution.
Time Tortoise Unit Testing with xUnit.NET
Build a fresh copy of the source
To reproduce the broken state without affecting my working environment, I cloned a fresh copy of Time Tortoise from GitHub, and built the solution. As expected, most of my unit tests were not discovered. I did see one test, SignalRServer_StartsAndShutsDownWithoutError. That test isn’t affected by the problem because it’s in a .NET Classic Desktop class library.
Update the target framework
The xUnit.net Getting Started page has instructions for creating a class library from the command line using the dotnet net classlib command. It also has this interesting comment about .NET Core vs. .NET Standard:
The result of this project template creates a .NET Standard class library. We actually want a .NET Core class library (since we’re writing .NET Core tests).
Comments on an xUnit GitHub bug called How to test with xUnit libraries that target .NET Standard 1.5? also imply that .NET Core is the correct framework to use for xUnit class libraries. I know it’s not mandatory to use .NET Core, since I have been using .NET Standard class libraries for xUnit. But since the documentation recommends .NET Core, I decided to see if it would work better.
Both .NET Standard and .NET Core have simplified csproj file formats compared to .NET Classic. For .NET Standard projects, the following PropertyGroup appears in the csproj:
<PropertyGroup>
<TargetFramework>netstandard1.4</TargetFramework>
</PropertyGroup>
And for .NET Core, it’s defined like this:
<PropertyGroup>
<TargetFramework>netcoreapp1.1</TargetFramework>
</PropertyGroup>
That’s the only difference between the csproj files for the two frameworks. So I manually updated the TargetFramework to netcoreapp1.1 (the version of .NET Core I’m currently using) for my three .NET Standard xUnit class libraries.
Build again, and run tests
After updating the target framework settings, I built the solution again. Amazingly, all of my tests were discovered, without any manual DLL dependency copying. Even better, they all passed.
Code coverage
As part of my unit testing process, I also track code coverage, so I know where I’m missing tests. Unfortunately, this is where the fix to the unit test dependency problem started to come apart.
When I ran code coverage using my updated unit test projects, I only got coverage results for tests in .NET Classic class libraries. The Assembly Binding Log Viewer reported several familiar binding errors, which I could of course track down using the steps I have previously documented. But if I’m going to have to copy a bunch of DLLs again, there’s not any benefit in going with this .NET Core-based xUnit approach.
Nevertheless, for research purposes, I tried converting my three .NET Standard test projects to .NET Core in my original working directory, which already contained all of the required dependent DLLs. Running code coverage in that environment caused almost all of the tests to either fail or not be discovered. The ones that failed reported this error:
Message: System.IO.FileNotFoundException : Could not load file or assembly ‘Microsoft.VisualStudio.CodeCoverage.Shim
A web search for this message leads to a .NET CLI bug report that implies that code coverage only works for .NET Standard projects, not .NET Core.
Back to .NET Standard
After this week’s experiment, it seems clear that for the environment I have — .NET Core/Standard, xUnit.NET, and Visual Studio built-in code coverage — manually copying dependent DLLs is a reasonable approach, despite the manual work involved. Maybe there’s a better solution out there, but the one I’m using is the best I have been able to find.