It might have taken 4 years, but I think it’s finally time to declare the xUnit.net test runner for ReSharper ready for 1.0 status.
This is a big update, and a long post (but there’s screenshots!) so let’s cut straight to the chase. If you want a download link, go straight to CodePlex and grab it. If you’re interested in what’s new, let’s start with a bullet list:
Where’s the ReSharper 8.0 EAP support? The code’s there, it’s ready to go. But the current EAP builds have an issue that prevents the test runner being loaded unless it’s in the product install directory, so it can’t work for plugins right now. This should be fixed soon.
Right. Screenshot time.
PropertyData attribute support
The PropertyDataAttribute tells xunit to get data for a Theory parameterised test by calling the named property’s getter. It relies on a magic string. And we all know that magic strings break when you rename the thing its referring to. Wouldn’t it be nice if ReSharper could help here?
And of course, it can. This release tries to add a reference between the string value of the attribute declaration and the property it’s referring to. So, if ReSharper can’t find a property with the same name as the string, it’s displayed as an error:
If it can find it, then it happily takes part in the ReSharper world of loveliness – ctrl-click navigation, included in find usages, renamed during refactoring, and it even marks the property as in use.
And there’s code completion (oh look! That’s Visual Basic!):
And of course, it works with the overload that takes a type name as well as a property name:
Traits (categories) support
NUnit and MSTest both have a Category attribute. This takes a single string value, and ReSharper can use this to group tests, so you could, for example, run all “logging” tests at once. The xunit approach is a bit different, though. Instead of a Category attribute, xunit has the more general Trait attribute, that takes two string parameters, a name and a value. It’s trivial to model category with this, just use a name of “category”, but what do you do with the other trait values?
This answer is of course obvious, especially in hindsight (this post on how the TFS team had implemented it in the Visual Studio test runner was the light bulb moment for me). If you have two strings, and need to make one, just concatenate them. So, name/value traits are formatted as “name[value]” and this is treated as the ReSharper category. The name “category” gets special treatment. As long as the name is “category”, regardless of case, only the value is used. This allows for usage such as [Trait(“category”, “logging”)] to be grouped under the “logging” category, and something like [Trait(“owner”, “matt”)] to be grouped under the “owner[matt]” category. Note that only “category” gets the special, case insensitive, treatment – “owner” is not the same as “Owner”.
And there is code completion here too, on both the name and value (yes, more VB!):
This has been one of the longest outstanding issues. The RunWith attribute allows you to change how the tests for a particular class are run – you get to take over the test discovery process. The default process looks for all methods that are decorated with a Fact or Fact derived attribute, but you could create a test class command that returns tests that being with “Test” or “Should”. Or one that calls a method in the class to get the tests to execute, or which runs the tests in a specific order.
The tricky thing here is that the ReSharper plugin can only go on what’s in the source code – it can’t run arbitrary code to find out what methods (if any!) should be tests. And it might be a random process, that changes each time the code is run. Sounds a lot like Theory support, in fact.
When the plugin encounters a class decorated with the RunWith attribute, it marks the class as being a test class, but it doesn’t mark any methods as test methods, even if they’re decorated with the Fact attribute – this could be a red herring if the custom RunWith test discovery is based on method name.
When the class is run, the methods are dynamically added to the test runner results window. Double clicking on the test will navigate to the appropriate method, and individual methods can be run and debugged just as with normal methods and theories. (The PrioritizedFixture shown here comes from the xunit samples)
There have been live templates in the release for years, but they’ve been a bit neglected – they were still in the old ReSharper 4.1 xml format! I’ve given them a bit of love and updated and added to them, and they’re now a first class part of the plugin.
The best bit is that you don’t have to do anything to get them – the plugin adds them to the settings system automatically. Just go to the Templates Explorer and see all the new templates tagged “xunit” in the C# section.
Wherever possible, the shortcuts have been chosen to be mnemonics of the assert, “ae” is Assert.Equal, “at” Assert.True, “athr” Assert.Throws, “adnt” Assert.DoesNotThrow, and so on. There are a few more for creating test methods, including “fact” to create a method decorated with [Fact], “fa” expands to [Fact] (Fact Attribute), xtc expands to an Xunit Test Class, and so on. Check them out in the Templates Explorer.
If you don’t like the shortcuts, or the code that’s generated, simply edit them. The changes are saved in your settings by default, and don’t override the shipped templates. (Although if you’re editing them, why not send me a pull request with the changes?)
But if you really don’t like the templates, you can disable them completely, by going to the ReSharper –> Manage Options dialog and unchecking the “xunitcontrib templates” settings layer. This removes them from the system completely.
External annotations have been in the zip file for years, too, also neglected. ReSharper’s analysis engine is very clever, but it can be even smarter if you give it more context by annotating your code with certain attributes. You can tell ReSharper when a return value or parameter will be null or not null, or if the method will throw an exception if a certain parameter is null, not null, true or false. You can even say that a lambda is being used immediately, rather than stashed away for later use, which limits the scope of a closure, and stop ReSharper showing you the dreaded “access to modified closure” or “implicitly captured closure” warnings.
And if you don’t have access to the source to add these attributes? Then you create an external annotations file, which maps annotation attributes to type members. The annotations for xunit have been updated to xUnit 1.9.1, and are now automatically installed.
Aborting test runs
This was something I hadn’t noticed about the test runner. When the tests are running, hit the stop button. The button changes to a hand icon, but the tests keep running. Hitting the button a second time stops the tests immediately. What’s going on here?
The first hit signals to the test runner process that you want to abort, and waits for the test runner to comply. If it doesn’t stop in good time, and you hit it again, it forcibly kills the test runner process.
Turns out, the xunit runner wasn’t paying any attention to the signal to stop, so the only way to abort a test run was to kill the process. And then the shadow copy cache didn’t get deleted, leaving folders lying around in your temp directory.
This release now plays nicely with the stop button, and will stop after the current test is run. If you forcibly end the run, the cache gets cleaned up correctly.
And finally, because no plugin should be without them, the plugin will now check for updates, and let you know if one is available.
I need to test these changes in the standalone dotCover test runners, and get a 1.0 release out there. And then, it’s full steam onto version 2.0, which will be all about the xunit 2.0 rewrite.
Until then, go get the 1.0 version, and go write lots of lovely xunit tests.