I posted a little while ago that ReSharper had updated to 4.5.1, and that the test runner in xunitcontrib worked just fine with it, even though it was actually compiled against 4.5.
Except for one little edge case – all of the xunitcontrib tests themselves.
That was a little… perturbing.
ReSharperRunnerLoggerTests.ExceptionThrownCallsReportsExceptionAndFinishesClassTask : Failed
System.IO.FileLoadException: Could not load file or assembly
Culture=neutral, PublicKeyToken=1010a0d8d6380325' or one of its dependencies.
The located assembly's manifest definition does not match the assembly reference.
(Exception from HRESULT: 0x80131040)
So what’s happening here?
The test runner plugin, compiled against ReSharper 4.5, is running (successfully) in the ReSharper 4.5.1 test runner. It is trying to execute its own test assembly, which is also compiled against ReSharper 4.5 (aka 4.5.1231.7). This test assembly is unable to find the ReSharper 4.5 assemblies, which are deployed into the same directory as the test assembly (I checked. Several times). And just to reiterate – the test runner plugin had no such assembly loading issues.
Surprisingly, this does make sense. It just took me a while to figure out.
Let’s start with the easy stuff. The test runner plugin is working due to plain old assembly redirects. ReSharper runs tests out of process, in an executable called JetBrains.ReSharper.TaskRunner.exe. If you look in its config file, you’ll see a ton of assembly redirects to map all ReSharper assemblies from 188.8.131.52 and above to 4.5.1274.1 (ReSharper 4.5.1). So, when xunitcontrib is used to run some tests, the exe fires up, loads the plugin, redirects my dependencies to the newer versions and all is well.
(xunitcontrib also runs in the devenv.exe process. A quick look at the devenv.exe.config file sees another ton of redirects. I know it’s the correct way to do this, but it does feel a bit crufty stomping all of that into someone else’s config…)
That explains why the test runner works. Why don’t the tests?
Thanks to the very same redirects that make the test runner work.
Tests are run in the same JetBrains.ReSharper.TaskRunner.exe as the plugin, so the same redirects apply. The problem starts because the ReSharper test runner sets up some custom assembly resolving handling, meaning that when the CLR tries to find a ReSharper assembly, the test runner can point it to the ReSharper installation directory. But because the tests run in a new AppDomain, they don’t get this custom assembly lookup, and so can’t find the new assemblies. Annoyingly, it reports this as being unable to load the originally requested version of the assembly (the version which is in the same directory, causing lots of confusion).
There are two solutions to this.
Obvious: recompile to 4.5.1.
Simple: add an empty app.config file. When the new AppDomain is created, it uses this config file, so no more redirects and no more failing tests.
So, if you ever get stuck trying to dogfood a test runner to run its own tests in another AppDomain within the same process while redirecting assembly references - this post’s for you.