Assembly binding redirects and pathological edge cases

by Matt 27. August 2009 19:07

I posted a little while ago that ReSharper had updated to 4.5.1, and that the test runner in xunitcontrib worked just fine with it, even though it was actually compiled against 4.5.

Except for one little edge case – all of the xunitcontrib tests themselves.

That was a little… perturbing.

ReSharperRunnerLoggerTests.ExceptionThrownCallsReportsExceptionAndFinishesClassTask : Failed

System.IO.FileLoadException: Could not load file or assembly
'JetBrains.ReSharper.TaskRunnerFramework, Version=4.5.1231.7, 
Culture=neutral, PublicKeyToken=1010a0d8d6380325' or one of its dependencies.
The located assembly's manifest definition does not match the assembly reference.
(Exception from HRESULT: 0x80131040) 

So what’s happening here?

The test runner plugin, compiled against ReSharper 4.5, is running (successfully) in the ReSharper 4.5.1 test runner. It is trying to execute its own test assembly, which is also compiled against ReSharper 4.5 (aka 4.5.1231.7). This test assembly is unable to find the ReSharper 4.5 assemblies, which are deployed into the same directory as the test assembly (I checked. Several times). And just to reiterate – the test runner plugin had no such assembly loading issues.

Surprisingly, this does make sense. It just took me a while to figure out.

Let’s start with the easy stuff. The test runner plugin is working due to plain old assembly redirects. ReSharper runs tests out of process, in an executable called JetBrains.ReSharper.TaskRunner.exe. If you look in its config file, you’ll see a ton of assembly redirects to map all ReSharper assemblies from and above to 4.5.1274.1 (ReSharper 4.5.1). So, when xunitcontrib is used to run some tests, the exe fires up, loads the plugin, redirects my dependencies to the newer versions and all is well.

(xunitcontrib also runs in the devenv.exe process. A quick look at the devenv.exe.config file sees another ton of redirects. I know it’s the correct way to do this, but it does feel a bit crufty stomping all of that into someone else’s config…)

That explains why the test runner works. Why don’t the tests?

Thanks to the very same redirects that make the test runner work.

Tests are run in the same JetBrains.ReSharper.TaskRunner.exe as the plugin, so the same redirects apply. The problem starts because the ReSharper test runner sets up some custom assembly resolving handling, meaning that when the CLR tries to find a ReSharper assembly, the test runner can point it to the ReSharper installation directory. But because the tests run in a new AppDomain, they don’t get this custom assembly lookup, and so can’t find the new assemblies. Annoyingly, it reports this as being unable to load the originally requested version of the assembly (the version which is in the same directory, causing lots of confusion).

There are two solutions to this.

Obvious: recompile to 4.5.1.

Simple: add an empty app.config file. When the new AppDomain is created, it uses this config file, so no more redirects and no more failing tests.

So, if you ever get stuck trying to dogfood a test runner to run its own tests in another AppDomain within the same process while redirecting assembly references - this post’s for you.

Tags: , , , ,

Separate naming styles for tests in ReSharper

by Matt 1. August 2009 18:38

The aim of my last post was to let folks know that the existing release of xunitcontrib works just fine with the recently released ReSharper 4.5.1, oh-and-by-the-way a nice new feature is supported out of the box.

But it occurs to me that the nice new feature is interesting in and of itself, and I haven’t seen it mentioned much elsewhere. It allows you to specify different naming conventions for test and normal class and method names, easily adding support for BDD style naming. And it works with nunit, mstest and (with xunitcontrib installed) xunit.

So, in case you skipped the last post because you’re not interested in xunit or xunitcontrib, allow me to do a codinghorror and shamelessly link to myself for instructions and pictures. It talks a lot about xunit, but it applies to all test providers.

And if have already read it, please forgive the repeat and don’t bother reading this one.


xunitcontrib + ReSharper 4.5.1

by Matt 31. July 2009 16:03

Just the other day, JetBrains released ReSharper 4.5.1. There are no major changes; it’s a maintenance release.

And the good news is, the latest version of xunitcontrib (0.3.1) works just fine with it. No changes. Nothing.

But there’s one nice, new, quiet, little feature.

ReSharper 4.5 added naming style rules, such that you would get a visible squiggly warning when classes weren’t named InCamelCase and variables weren’t lowerCaseCamel.

Too bad for those upstart BDD folks that like to ape Ruby with Significant_use_of_the_underscore_character when naming their tests (“underscore porn”). These guys would get a warning under their test methods.

ReSharper warns that test method does not match naming standard

We can fix this in 4.5.1 by adding a user defined naming rule. Go to the ReSharper options dialog, and select Naming Style from the tree on the left hand side. Then select Add in the bottom half of this dialog. We get a new window:

The ReSharper user defined naming rule dialog

We want to set the rule just as in the picture – give it a description, select Test type and Test method (property) and then select First_upper.

And now our Interestingly_named_methods are no longer marked as being non-standard, and our “normally” named tests are prompted for renaming.


Cool. But what about your existing, non-BDD style tests? You don’t have to rename those. Simply add a new naming style to the rule you’ve just created. So not only does the rule for test elements enforce Uppercase_with_underscores, it also enforces UpperCamelCase. Very handy.

And in case you missed why I’m pointing this out, ReSharper is smart enough to realise that this rule applies to any test identified by a test provider, such as xunitcontrib. Those examples above are xunit Facts.

And the best bit? I didn’t have to do any work.

Tags: , ,

xunitcontrib 0.3.1 bug fix release

by Matt 15. July 2009 19:51

I’ve just released a minor bug fix release to the xunitcontrib ReSharper runner.

As part of the recent rewrite to support version independence, I inadvertently broke the current directory. If you know what I mean. Previously, ReSharper was setting the current directory to be the location of the test assembly. Because I’m now handing off to the xunit runner API, ReSharper’s code was no longer being used, and the current directory remained pointing to the location of the plugin assembly.

Since this was quite a major change in behaviour, and very much a blocking issue, I wanted to get a fix out as quickly as possible. Hence release 0.3.1. Nothing else has changed, so if you’re affected by this issue, please go get the latest version.

And thanks for the feedback!

Tags: ,

xunitcontrib 0.3 – external annotations and live templates

by Matt 9. July 2009 20:33

As part of the latest release of xunitcontrib, I’ve added two new features, support for ReSharper’s External Annotations and Live Templates.

External Annotations

ReSharper performs some static analysis on your code, and displays a blue squiggly error indicator over anything it thinks will be a problem. One great example is possible System.NullReferenceExceptions – using an object which may be null.

Possible System.NullReferenceException is flagged 

One thing you might notice from that screenshot is that we’ve got an Assert.NotNull in there. This method throws an exception if the “otherFlagsAttribute” is null, meaning we can’t get a NullReferenceException. But ReSharper doesn’t know that, and so we still get the warning.

As I understand it, ReSharper can only analyse your source code, and not any compiled assemblies. So how do we tell it that Assert.NotNull ensures that we don’t get a NullReferenceException?

Well, ReSharper makes available an assembly called JetBrains.Annotations.dll that provides a number of attributes that can handle this scenario (plus a whole bunch more, like not null parameters, or marking parameters as string format specifiers). Handy, but a bit intrusive to your code, and not much use if you’ve not got the code available to change.

And that’s where External Annotations come in. You can now unobtrusively apply these attributes to the code through xml files. (And the ReSharper guys have very nicely produced files for the entire .net framework…)

So, that’s what this feature is. It adds the xml file telling ReSharper that xunit’s Assert.NotNull is an assert method that will stop execution of the method if the parameter is null. Similarly, there’s annotations for Assert.Null, Assert.True and Assert.False.

(Interestingly, it’s easy to concoct an example for NotNull and also for True or False (simply create an expression that is always false or always true) but I can’t create an example for Assert.Null. Answers on a postcard, please…)

To install, simply drop the xunit.xml file into ProgramFiles\JetBrains\ReSharper\v4.x\bin\External Annotations.

Live Templates

Live Templates are very much like Visual Studio’s built in snippets feature (I guess I ought to look into porting this to snippets…) but with a bit more oomph. They allow you to type an identifier that automatically expands into something useful. For example, the Live Template provided expands the word “fact” into:

"fact" live template expanded as a fact method

Which has the method name highlighted ready to edit, and when you hit enter, places the cursor into the body of the method. It also automatically adds any using statements necessary.

These are very useful, especially with the smart Intellisense shown for the assert templates:

Assert.Equal Live Template with smart intellisense

I’ve actually released two versions of this. One which uses acronyms such as “ae” and “ann” to represent “Assert.Equal” and “Assert.NotNull”, and another that uses similar acronyms, but beginning with “x” (“xe”, “xnn”). This is to avoid a collision with Live Templates for a certain other unit testing framework that I use.

Here’s a list of the Live Templates included (the words wrapped in dollar signs like $this$ are macros, and are quickly edited by pressing tab, and usually display some sort of useful Intellisense):

Shortcut #1 Shortcut #2 Expands to
ae xe Assert.Equal($expected$, $actual$)
at xt Assert.True($value$)
af xf Assert.False($value$)
ac xc Assert.Contains($expected$, $actual$)
an xn Assert.Null($value$)
ann xnn Assert.NotNull($value$)
athr xthr Assert.Throws<$T$>()
aiaf xiaf Assert.IsAssignableFrom<$T$>($object$)
ait xit Assert.IsType<$T$>($object$)
fact fact Creates an attributed test/fact method
fa fa [Fact]
theory theory Creates an attributes theory method
ta ta [Theory]
ida ida [InlineData()]

Note that not all of the assert methods are there. Only the most common have been added. And I haven’t added Assert.Same, namely because “as” is a poor acronym to go for.

This is a little trickier to install. In Visual Studio, go to ReSharper –> Live Templates, then click the button to “Mount storage” and navigate to and select the live template file of your choice (“ae” or “xe” format).

If you mount the file, then any changes are saved back to this file. If you import it, then any changes are kept in a copy of the config file. I prefer mounting the file, as it gives me more flexibility to change the templates and pass the changes around.

So that covers external annotations and live templates. As ever, if you have any bugs or comments, please use the Issues and Discussion pages of the CodePlex project site. And feel free to get in touch on Twitter, too (via @citizenmatt or #xunitcontrib).

Tags: ,

My favourite xunitcontrib release yet

by Matt 9. July 2009 20:18

Yep. It’s xunitcontrib release time again, and this one’s a pretty major one. If you use the ReSharper runner, you really want to download this one.

(If you want to cut to the chase and just get the binaries and install instructions, then check out the release page on CodePlex.)

As usual, it’s a ReSharper focussed release, with improvements to the test runner and the addition of External Annotations and Live Templates. I’ll look at these last two in another post, because I want to focus on the big ticket item today – the runner.

So what’s the big news?

Well, if you’ll forgive a little enthusiastic hyperbole:

A self congratulatory twitter post announcing version independent test runner

Version independence.


Up until now, the ReSharper test runner would run your tests with the 1.1 released version of xunit.dll only. If your test project is using a different version, such as the new CTP of 1.5, you’ll get a cryptic error message, and no tests would run.

One significant rewrite later, and it’s now using the same version independent runner API that the console and msbuild runners use.

In short, you can now use any version of xunit you like.

Cool, and very much a big deal.

But rather embarrassingly, the bit which pleases me most is a simple side effect.

One of the nice things that xunit does is randomise the order in which tests are run. And because I’m now letting xunit handle everything, we get that for free:

The ReSharper test runner running tests in a random order

(I really should have done that as an animated gif; it’s nice to see in action.)

So that’s the Big New Feature. What else is there? This is a small one, but I like it. Nested exceptions are now formatted nicer. Before:

nested execeptions formatted as a single exception


nested execptions formatted as separate exceptions

And give or take a couple of things, that’s pretty much it for user-visible changes. But there are a couple of other things I want to point out though.

Firstly, that breathless Twitter post above came from a deep dive into the ReSharper test runner framework. Amusingly (after having worked on this project for the past 4 or 5 months) I finally understand how all of this works. And I’ve been heavily commenting the code to explain what’s going on. I hope it can be of use to someone else who wants to write a test runner.

(Yes, that’s right, I just made code comments a feature.)

I also have another post brewing about my developer conscience kicking in and pointing out that I could actually write unit tests for the newly restructured code. (Short version: TDD rocks + I finally get BDD.)

But it’s not all roses. There’s still no installer, and there’s truly terrible support for theory style tests (and I mean terrible. You’ll see output and exceptions, but not together. And you won’t know which test row caused the output, especially since they’re now run in random order too)

So what’s next?

Well, in part, that’s up to you. Check out the Issues on CodePlex and vote for what you want. Leave a comment here or on Twitter (via @citizenmatt or #xunitcontrib). But I really want to do something about those theory tests. I’m not sure what yet, though.

Right, that’s enough. Go and download it already.

Tags: ,

Announcing xunitcontrib 0.2 – Resharper 4.5 support

by Matt 14. April 2009 17:24

I’ve just released 0.2 of the xunitcontrib project (actually thought I’d released it over the weekend. Hadn’t pressed the right button. Oops – it’s there now.)

This release is primarily to support the RTM release of Resharper 4.5 (with a couple of smaller bug fixes too). It runs just like the old one, but with one added feature specifically for 4.5 – support for Resharper’s Solution Wide Code Inspections. When this is turned on, Resharper will highlight any public methods that are not used as part of a project. And of course, test methods are public methods that are not used by any other part of a project. One implementation of the wonderfully named IsTestStuff, and we’re all sorted.

Check out the downloads page for more details, and instructions on how to install.

And if you’ve got any comments, issues or ideas, please post them to the CodePlex site or leave a comment here, or via twitter, either directly to me (@citizenmatt) or via the #xunitcontrib hashtag.

Tags: ,


Month List


Comment RSS