ReSharper runner for 2.0

by Matt 2. May 2014 05:55

Here’s two nice things to end a Friday with.

Firstly, I’ve just pushed a new, 2.0.0 ALPHA pre-release version of the runner for ReSharper to the extension gallery. It adds support for 2.0 beta 2. You can get it from ReSharper’s Extension Manager. Make sure you select the “pre-release” drop down in the Extension Manager first.

Secondly, the project is no longer hosted on CodePlex, but has moved to GitHub, and lives under the xunit org. All issues and pull requests should go there please. 2

You should consider this an alpha version, because, well, it is. It works, I’ve got a bunch of tests to prove it, but hey, it’s all new and different, so if you find any issues, please, please, please, let me know (twitter or, even better, GitHub. Did I mention the project’s moved?) In fact, any kind of feedback is appreciated, good or bad.

Thanks to the excellent support built into xunit2 by Brad and Jim, the new runner can handle both your xunit1 and xunit2 tests. Everything works as you’d expect – tests are highlighted in the editor, and you can run and debug them as normal.

However, this is an alpha build. There are couple of important caveats:

  • Parallelisation is DISABLED in this release. This is due to how xunit2 handles errors. Currently, there is no way for me to gracefully handle errors thrown when a class or collection fixture throws an exception. Hopefully this can be resolved, and I’ll enable parallelisation.
  • Test discovery in the editor is currently handled by xunit1. This needs migration, but is a complex beast. It means that currently, anything more interesting than plain Fact or Theory based tests might not work. Test discovery in the runner is handled by xunit2, hence the weasel word “might”. Fixing this is my next priority.
  • It currently only supports 8.2, as this is what I’ve been using to build and test. I’m not sure yet if I’ll add support for 8.1 back. Leave feedback on GitHub if you really want it – but leave a reason, too. After all, 8.2 is a free upgrade!
  • Annotations and live templates haven’t been updated yet, or even tested. They might work, they might not – so your methods might not be highlighted as in use, and the templates might not work (theory definitely won’t – it’s in a different namespace)

Moving to GitHub

I think it’s fair to say that xunitcontrib had a good run on CodePlex, but, well, GitHub is better. It’s a much nicer experience, everything is just easier, and well, GitHub won.

A very nice feature on GitHub is organisations. Brad and Jim have very generously invited the ReSharper runner to live under the xunit org. It doesn’t change the ownership, copyright or license of the project ( is an Outercurve project, but the ReSharper runner is mine), it’s simply a nice place to live.

The name of the project has also changed. It always was more of a ReSharper project than a contrib project, so the name is now “resharper-xunit” to reflect this. I’ll still be using “xunitcontrib” as the name of the ReSharper packages, though.

What’s next?

Time to improve on the support. First item on the list is proper xunit2 based test discovery. I’ve opened a PR to track xunit2 progress, and more issues to track new functionality.

Please test it! Get it from the extension manager, and put it through its paces. And if you find any problems, report them!

Tags: , , ,


xunitcontrib-resharper 1.0: Stick a fork in it - it’s done

by Matt 4. April 2013 04:27

It might have taken 4 years, but I think it’s finally time to declare the test runner for ReSharper ready for 1.0 status.

This is a big update, and a long post (but there’s screenshots!) so let’s cut straight to the chase. If you want a download link, go straight to CodePlex and grab it. If you’re interested in what’s new, let’s start with a bullet list:

Where’s the ReSharper 8.0 EAP support? The code’s there, it’s ready to go. But the current EAP builds have an issue that prevents the test runner being loaded unless it’s in the product install directory, so it can’t work for plugins right now. This should be fixed soon.

Right. Screenshot time.

PropertyData attribute support

The PropertyDataAttribute tells xunit to get data for a Theory parameterised test by calling the named property’s getter. It relies on a magic string. And we all know that magic strings break when you rename the thing its referring to. Wouldn’t it be nice if ReSharper could help here?

And of course, it can. This release tries to add a reference between the string value of the attribute declaration and the property it’s referring to. So, if ReSharper can’t find a property with the same name as the string, it’s displayed as an error:

Cannot resolve symbol for PropertyDataAttribute

If it can find it, then it happily takes part in the ReSharper world of loveliness – ctrl-click navigation, included in find usages, renamed during refactoring, and it even marks the property as in use.

Property data highlighting usages

And there’s code completion (oh look! That’s Visual Basic!):

Code completion with PropertyDataAttribute

And of course, it works with the overload that takes a type name as well as a property name:

Code completion on property data attribute with a different type

Traits (categories) support

NUnit and MSTest both have a Category attribute. This takes a single string value, and ReSharper can use this to group tests, so you could, for example, run all “logging” tests at once. The xunit approach is a bit different, though. Instead of a Category attribute, xunit has the more general Trait attribute, that takes two string parameters, a name and a value. It’s trivial to model category with this, just use a name of “category”, but what do you do with the other trait values?

This answer is of course obvious, especially in hindsight (this post on how the TFS team had implemented it in the Visual Studio test runner was the light bulb moment for me). If you have two strings, and need to make one, just concatenate them. So, name/value traits are formatted as “name[value]” and this is treated as the ReSharper category. The name “category” gets special treatment. As long as the name is “category”, regardless of case, only the value is used. This allows for usage such as [Trait(“category”, “logging”)] to be grouped under the “logging” category, and something like [Trait(“owner”, “matt”)] to be grouped under the “owner[matt]” category. Note that only “category” gets the special, case insensitive, treatment – “owner” is not the same as “Owner”.

Trait showing category

And there is code completion here too, on both the name and value (yes, more VB!):


RunWith support

This has been one of the longest outstanding issues. The RunWith attribute allows you to change how the tests for a particular class are run – you get to take over the test discovery process. The default process looks for all methods that are decorated with a Fact or Fact derived attribute, but you could create a test class command that returns tests that being with “Test” or “Should”. Or one that calls a method in the class to get the tests to execute, or which runs the tests in a specific order.

The tricky thing here is that the ReSharper plugin can only go on what’s in the source code – it can’t run arbitrary code to find out what methods (if any!) should be tests. And it might be a random process, that changes each time the code is run. Sounds a lot like Theory support, in fact.

When the plugin encounters a class decorated with the RunWith attribute, it marks the class as being a test class, but it doesn’t mark any methods as test methods, even if they’re decorated with the Fact attribute – this could be a red herring if the custom RunWith test discovery is based on method name.

When the class is run, the methods are dynamically added to the test runner results window. Double clicking on the test will navigate to the appropriate method, and individual methods can be run and debugged just as with normal methods and theories. (The PrioritizedFixture shown here comes from the xunit samples)


Live Templates

There have been live templates in the release for years, but they’ve been a bit neglected – they were still in the old ReSharper 4.1 xml format! I’ve given them a bit of love and updated and added to them, and they’re now a first class part of the plugin.

The best bit is that you don’t have to do anything to get them – the plugin adds them to the settings system automatically. Just go to the Templates Explorer and see all the new templates tagged “xunit” in the C# section.


Wherever possible, the shortcuts have been chosen to be mnemonics of the assert, “ae” is Assert.Equal, “at” Assert.True, “athr” Assert.Throws, “adnt” Assert.DoesNotThrow, and so on. There are a few more for creating test methods, including “fact” to create a method decorated with [Fact], “fa” expands to [Fact] (Fact Attribute), xtc expands to an Xunit Test Class, and so on. Check them out in the Templates Explorer.

If you don’t like the shortcuts, or the code that’s generated, simply edit them. The changes are saved in your settings by default, and don’t override the shipped templates. (Although if you’re editing them, why not send me a pull request with the changes?)

But if you really don’t like the templates, you can disable them completely, by going to the ReSharper –> Manage Options dialog and unchecking the “xunitcontrib templates” settings layer. This removes them from the system completely.

Manage settings layer showing custom templates layer

External Annotations

External annotations have been in the zip file for years, too, also neglected. ReSharper’s analysis engine is very clever, but it can be even smarter if you give it more context by annotating your code with certain attributes. You can tell ReSharper when a return value or parameter will be null or not null, or if the method will throw an exception if a certain parameter is null, not null, true or false. You can even say that a lambda is being used immediately, rather than stashed away for later use, which limits the scope of a closure, and stop ReSharper showing you the dreaded “access to modified closure” or “implicitly captured closure” warnings.

And if you don’t have access to the source to add these attributes? Then you create an external annotations file, which maps annotation attributes to type members. The annotations for xunit have been updated to xUnit 1.9.1, and are now automatically installed.

Unreachable code due to annotations

Aborting test runs

This was something I hadn’t noticed about the test runner. When the tests are running, hit the stop button. The button changes to a hand icon, but the tests keep running. Hitting the button a second time stops the tests immediately. What’s going on here?

The first hit signals to the test runner process that you want to abort, and waits for the test runner to comply. If it doesn’t stop in good time, and you hit it again, it forcibly kills the test runner process.

Turns out, the xunit runner wasn’t paying any attention to the signal to stop, so the only way to abort a test run was to kill the process. And then the shadow copy cache didn’t get deleted, leaving folders lying around in your temp directory.

This release now plays nicely with the stop button, and will stop after the current test is run. If you forcibly end the run, the cache gets cleaned up correctly.

Update notifications

And finally, because no plugin should be without them, the plugin will now check for updates, and let you know if one is available.

What’s next?

I need to test these changes in the standalone dotCover test runners, and get a 1.0 release out there. And then, it’s full steam onto version 2.0, which will be all about the xunit 2.0 rewrite.

Until then, go get the 1.0 version, and go write lots of lovely xunit tests.

Tags: , ,

xunitcontrib-resharper 0.7 - ReSharper 7.1 and stuff

by Matt 20. November 2012 03:39

ReSharper has just released 7.1, so I think it’s only fair to update the test runner to support it.

And it would be mean not to throw in a couple of bug fixes while I’m at it.

Firstly, there’s a fix to support filtering out test usages when using ReSharper’s Find Usages command. Frankly, I thought this was working ages ago, but it looks like I broke it some time back. Sorry folks.

When in the Find Results window, displaying the usages of a type, method, property, etc. you can filter out different types of usage – read/write, invocation, usage in attributes or documentation, and, what’s interesting here, usages in tests. This means you can hide any usage of a type from a test method or class. Simply toggle the “Show Unit Test Usages” menu item in the filter, and the Results window will hide and show the test usages (note that the test usages are using the test icon that appears next to the test method in the code editor).


The second bug fix relates to Theory data rows. uses class and method names as a means of identifying a test. This is guaranteed to be a unique value, as the test method doesn’t run more than once. For Theory parameterised tests, xunit uses the class and method name, but also uses a display name, which includes the parameter values, such as “ShouldBeUppercase(s: SAUSAGES)”.

There is a problem here. If you have two rows of data with the same value (“SAUSAGES”), the test runner treats these as the same test. This is a case of “doctor, doctor, it hurts when I do this”. Don’t do this. You’re passing in the same data, you’ll get the same results, it doesn’t matter if it looks like there’s only one test.

Or so I thought.

What happens if the data being passed in is an array of strings? xunit will call ToString on the parameter value, and come up with a display name like “ShouldBeUppercase(s: System.String[])”. Clearly, there are going to be collisions here. So now, the test runner keeps a track of names of tests it’s already run. If it tries to run the same test again, it changes the name, appending a call number, e.g. “ShouldBeUppercase(s: System.String[])” and “ShouldBeUppercase(s: System.String[]) [2]”, “… [3]” and so on.


And that’s it for this release. I smell a 1.0 version coming, finally. There are a couple of big ticket changes I want to make first. I want better support for PropertyDataAttribute (IntelliSense, find usages, ctrl-click navigation, etc) and InlineDataAttribute, and there’s a good memory optimisation I need to make for editing a large test file. If there’s anything else missing or broken, now’s a good time to let me know!

As ever, download it from Codeplex, install it with the handy batch file, let me know any issues.

Tags: , ,


xunitcontrib-resharper 0.6.1 - mostly bug fixes

by Matt 2. August 2012 02:42

Seems like this blog is becoming just an announcement service for new versions of my test runner for ReSharper. Bit of a maintenance release, this one. Here’s what it’s in:

RTM support for ReSharper 7.0 RTM:

The 0.6 release worked just fine with the RTM, but here’s the official build.

Bug fixes:

  1. The wrong dll was used when you switched between Debug and Release. Yikes!
  2. Adding a method to the end of a class, and then trying to run all tests in a class could cause the whole class to fail, as it tried to run a method called “???”
  3. If you used non-public test methods, you couldn’t double-click to navigate in the test sessions or test explorer views
  4. Exceptions were thrown when trying to parse a source file that contained static public properties and had a build action set to None. Rather a specific one, that.

The most interesting change is for placing tests in nested classes. When displaying a test method in the UI, it’s usually displayed as the class’s short name, e.g. For “Name.Space.ClassName”, you see “ClassName”. This scheme fails when you have nested classes, and multiple nested classes share a name, but have different parent classes. Like this:

public class AddressTests
  public class ToStringTests
    [Fact] public void ShouldReturnFormattedAddress() { /* ... */ }

public class PersonTests
  public class ToStringTests
    [Fact] public void ShouldReturnForenameAndSurname() { /* ... */ }

(Check out Phil Haack’s post that nicely describes why you might want to do this). The problem is that the test runner would display this:


making it very hard to distinguish which test was which. Now they display like this:


In other words, nested classes include the name of their parent class in the UI. I’m not exactly enamoured of the naming format; using a “+” to separate the class names isn’t very well known outside of Relection, but at least it gives you the chance to know which test is which. It might change in a future version - there’s an outstanding bug for ReSharper to improve the NUnit runner’s support for nested classes, and I’ll follow their lead.

Removed support for 5.1:

I started this project with ReSharper 4.1. I make that 7 releases (4.1, 4.5, 5.0, 5.1, 6.0 and the current 6.1.1 and 7.0). Going forward, I’m only going to support the last two major releases – i.e. 6.x and 7.x, so that means the current release has binaries for 6.1.1 and 7.0 RTM. You can still download the binaries to any of the old releases, and the source is still available in the repo, but they’re not going to see any new development.

So, there you have it. Go download it. If there’s a problem, report it at CodePlex, or give me a shout on Twitter.

Tags: , ,


xunitcontrib-resharper 0.6 - Theory support!

by Matt 5. July 2012 10:09


Yes, the latest release of the ReSharper test runner provides much better Theory support (theories are xunit’s parameterised tests). Up until now, the support has been so poor (i.e. non-existent) that it’s not always been possible to tell if a theory has failed, and if it has, which one did and why. It’s been the biggest issue and most requested feature since the project began.

Well, that’s all in the past now. Here’s the latest version running some Theory tests:


Yep, theories are now displayed as child nodes under the parent method, as you’d expect.

It provides support for any kind of Theory – data from attributes, properties, spreadsheets, databases and any other kind of custom Theory or multiple row producing custom Fact-based attribute you might like to write. It also works with dotCover to provide code coverage. (I’ll update the standalone dotCover runners shortly)

This release works with 6.1 and the current build of 7.0 (that is build 82). The download does not include runners for 6.0, 5.1 or earlier versions of ReSharper. Those versions are packaged up in an archive release. The idea is to only support the latest versions of the last two major releases of ReSharper.

As ever, download the release (PLEASE don’t forget to unblock the dlls), and if there are issues, let me know on codeplex or via twitter – @citizenmatt.

Tags: , ,

Running tests in dotCover 2.0 EAP

by Matt 18. April 2012 10:03

JetBrains recently released an EAP of dotCover 2.0. Up until now, dotCover required ReSharper to enable code coverage of unit tests – it integrated itself nicely in the ReSharper menus and UI, and would provide a coverage analysis of the code executed during a test run.


Starting with this 2.0 EAP, you no longer need ReSharper installed. DotCover provides support for its own test runner and test runner plugins. In fact, it comes bundled with a port of the unit test support and framwork from ReSharper, so you won’t actually be able to tell the difference.

Which means, of course, a new build of xunitcontrib to provide support for dotCover. This is a separate release to the ReSharper support – while the products have (pretty close) source code compatibility, there isn’t (yet?) binary compatibility.

Installation instructions are as simple as for the ReSharper plugin:

  1. Make sure you UNBLOCK the downloaded zip file (right click –> properties –> Unblock).
  3. Extract the zip and copy the xunitcontrib.runner.dotcover folder into C:\Program Files\JetBrains\dotCover\v2.0\bin\plugins (you will probably need to create the plugins folder)
  4. Restart Visual Studio.

Once installed, tests should be recognised, and you should now be able to run, debug and cover xunit tests.

Note that you DO NOT NEED this plugin if you have ReSharper (and the appropriate xunitcontrib version) installed. By default, dotCover still integrates with ReSharper, if it’s available.

If, however, you want to see what all the fuss is about, you can disable ReSharper’s unit test implementation, and use dotCover’s. Simply go to the ReSharper –> Options –> Unit Testing, uncheck “Enable Unit Testing”, and then go to the dotCover –> Options –> ReSharper Integration and uncheck “Hide all actions and tool windows related to unit tests in dotCover and use ReSharper’s implementation”. You will now be using the dotCover unit test implementation.

It’s going to look very familiar.


(And don’t forget to check out the “Colour Scheme” options if you’ve customised your Visual Studio colours. “Dark” makes SUCH a difference)


Tags: , ,

xunitcontrib-resharper 0.5.2 - ReSharper 6.1 support

by Matt 22. December 2011 12:35

ReSharper 6.1 has just been released. It’s better than 6.0, and it’s a free download, so go and get it. And then download the latest version of xunitcontrib to get support for your tests.

This version contains plugins for ReSharper 6.1, 6.0 and 5.1.3. I plan to only support the most recent revisions of the last two major versions, so that means 6.1 and 5.1.3. So, I’ll be moving 6.0 into the archive release very soon.

This is a small update. The important news is the support of ReSharper 6.1, but there was still chance to squeeze in a new feature.

ReSharper 6.0 added a nice little feature for running the derived instances of test methods defined in an abstract test method. To explain that a bit better, assume you’ve got an abstract base class that contains a test method:

public abstract class AbstractBaseClass
    public void TestMethod()
        // ...

and then a derived class:

public class DerivedClass : AbstractBaseClass
    // ...

ReSharper 6.x provides support for adding a marker next to AbstractBaseClass.TestMethod that gives a drop down menu allowing you to run TestMethod in one or all of the derived test classes. Something like this:


Unfortunately, 6.0 doesn’t provide enough support to get the naming correct everywhere in the UI. You can either specify the name as DerivedClass.TestMethod (which makes most sense when used in this drop down) or BaseClass.TestMethod (which reads better in unit test explorer and runner windows).

I’ve chosen to use BaseClass.TestMethod, because I think it’s more useful that the names are correct in the unit test explorer/runner windows (the built in nunit plugin uses DerivedClass.TestMethod). Annoyingly, this means the drop down isn’t terribly helpful:


The good news is that this all works correctly in 6.1.

So, what are you waiting for? Go upgrade.

Tags: , ,


Assembly binding redirects and pathological edge cases

by Matt 27. August 2009 19:07

I posted a little while ago that ReSharper had updated to 4.5.1, and that the test runner in xunitcontrib worked just fine with it, even though it was actually compiled against 4.5.

Except for one little edge case – all of the xunitcontrib tests themselves.

That was a little… perturbing.

ReSharperRunnerLoggerTests.ExceptionThrownCallsReportsExceptionAndFinishesClassTask : Failed

System.IO.FileLoadException: Could not load file or assembly
'JetBrains.ReSharper.TaskRunnerFramework, Version=4.5.1231.7, 
Culture=neutral, PublicKeyToken=1010a0d8d6380325' or one of its dependencies.
The located assembly's manifest definition does not match the assembly reference.
(Exception from HRESULT: 0x80131040) 

So what’s happening here?

The test runner plugin, compiled against ReSharper 4.5, is running (successfully) in the ReSharper 4.5.1 test runner. It is trying to execute its own test assembly, which is also compiled against ReSharper 4.5 (aka 4.5.1231.7). This test assembly is unable to find the ReSharper 4.5 assemblies, which are deployed into the same directory as the test assembly (I checked. Several times). And just to reiterate – the test runner plugin had no such assembly loading issues.

Surprisingly, this does make sense. It just took me a while to figure out.

Let’s start with the easy stuff. The test runner plugin is working due to plain old assembly redirects. ReSharper runs tests out of process, in an executable called JetBrains.ReSharper.TaskRunner.exe. If you look in its config file, you’ll see a ton of assembly redirects to map all ReSharper assemblies from and above to 4.5.1274.1 (ReSharper 4.5.1). So, when xunitcontrib is used to run some tests, the exe fires up, loads the plugin, redirects my dependencies to the newer versions and all is well.

(xunitcontrib also runs in the devenv.exe process. A quick look at the devenv.exe.config file sees another ton of redirects. I know it’s the correct way to do this, but it does feel a bit crufty stomping all of that into someone else’s config…)

That explains why the test runner works. Why don’t the tests?

Thanks to the very same redirects that make the test runner work.

Tests are run in the same JetBrains.ReSharper.TaskRunner.exe as the plugin, so the same redirects apply. The problem starts because the ReSharper test runner sets up some custom assembly resolving handling, meaning that when the CLR tries to find a ReSharper assembly, the test runner can point it to the ReSharper installation directory. But because the tests run in a new AppDomain, they don’t get this custom assembly lookup, and so can’t find the new assemblies. Annoyingly, it reports this as being unable to load the originally requested version of the assembly (the version which is in the same directory, causing lots of confusion).

There are two solutions to this.

Obvious: recompile to 4.5.1.

Simple: add an empty app.config file. When the new AppDomain is created, it uses this config file, so no more redirects and no more failing tests.

So, if you ever get stuck trying to dogfood a test runner to run its own tests in another AppDomain within the same process while redirecting assembly references - this post’s for you.

Tags: , , , ,

xunitcontrib + ReSharper 4.5.1

by Matt 31. July 2009 16:03

Just the other day, JetBrains released ReSharper 4.5.1. There are no major changes; it’s a maintenance release.

And the good news is, the latest version of xunitcontrib (0.3.1) works just fine with it. No changes. Nothing.

But there’s one nice, new, quiet, little feature.

ReSharper 4.5 added naming style rules, such that you would get a visible squiggly warning when classes weren’t named InCamelCase and variables weren’t lowerCaseCamel.

Too bad for those upstart BDD folks that like to ape Ruby with Significant_use_of_the_underscore_character when naming their tests (“underscore porn”). These guys would get a warning under their test methods.

ReSharper warns that test method does not match naming standard

We can fix this in 4.5.1 by adding a user defined naming rule. Go to the ReSharper options dialog, and select Naming Style from the tree on the left hand side. Then select Add in the bottom half of this dialog. We get a new window:

The ReSharper user defined naming rule dialog

We want to set the rule just as in the picture – give it a description, select Test type and Test method (property) and then select First_upper.

And now our Interestingly_named_methods are no longer marked as being non-standard, and our “normally” named tests are prompted for renaming.


Cool. But what about your existing, non-BDD style tests? You don’t have to rename those. Simply add a new naming style to the rule you’ve just created. So not only does the rule for test elements enforce Uppercase_with_underscores, it also enforces UpperCamelCase. Very handy.

And in case you missed why I’m pointing this out, ReSharper is smart enough to realise that this rule applies to any test identified by a test provider, such as xunitcontrib. Those examples above are xunit Facts.

And the best bit? I didn’t have to do any work.

Tags: , ,

xunitcontrib 0.3.1 bug fix release

by Matt 15. July 2009 19:51

I’ve just released a minor bug fix release to the xunitcontrib ReSharper runner.

As part of the recent rewrite to support version independence, I inadvertently broke the current directory. If you know what I mean. Previously, ReSharper was setting the current directory to be the location of the test assembly. Because I’m now handing off to the xunit runner API, ReSharper’s code was no longer being used, and the current directory remained pointing to the location of the plugin assembly.

Since this was quite a major change in behaviour, and very much a blocking issue, I wanted to get a fix out as quickly as possible. Hence release 0.3.1. Nothing else has changed, so if you’re affected by this issue, please go get the latest version.

And thanks for the feedback!

Tags: ,


Month List


Comment RSS