Well, I've found some more documentation. And it's got me a bit nervous.
The IFilter stuff is fine. I don't think that's changed at all since it was first created for NT4.
The problem is with the protocol handlers.
The MSN Toolbar docs point to this documentation for Sharepoint Portal Server SDK 2001. It, of course, talks about everything from a Sharepoint Portal Server 2001 point of view. And I of course have absolutely no idea what's going on. It's written from the point of view that you already know what each part of the system is, what they do, and how they talk. It's also 5 years old. Still, it links to the reference info for the required interfaces, so I guess it's a good start. There's even a sample, which seems to be the only place you can get the header files for ISearchProtocol.
Getting a little more modern, there's a slightly expanded version of the same docs for SharePoint Portal Server 2003 - only 3 years out of date. It has a nice little troubleshooting section, and provides a bit more detail on a couple of the interfaces to implement. There's also a bit of a reference section. But we're still talking SharePoint, not WDS.
Then there's the MSN Toolbar docs themselves. Again, it's all a bit everything-you-need-without-any-detail. But it does detail how you register a protocol handler (without telling you how to get the ISearchManger interface). The big problem is that it's for Windows Desktop Search when it was still called the MSN Desktop Search. I'm presuming the registration still works...
Things get really interesting when you go to the Windows Shell MSDN library page. Looking in the tree view on the left you'll see nodes for WDS 2.x and WDS 3.0. Only the 2.x version has a landing page. It's described as preliminary documentation, and only talks about IFilter. The interfaces listed in the reference section aren't terribly useful, either.
The WDS 3.0 section only has reference information, and lists 39 interfaces, a couple of which are also mentioned in the Sharepoint docs. Unfortunately, it's ISearchManager interface has completely different methods to the MSN Toolbar interface of the same name, and offers no way of registering protocols handlers. It looks like WDS has had quite an overhaul for version 3/Vista. This post on MSDN's forums seems to confirm that there's quite a difference.
And that's what's making me nervous. I've currently got WDS 3.0 Beta 2 running, but I think I'd be happier developing towards the 2.x version - because it looks more like the SharePoint stuff that I've got a sample for. Plus, I don't have the headers for the Vista stuff - I don't fancy downloading a beta version of the Platform SDK, especially since it contains the beta WinFx stuff.
(And it looks like I've got an out-of-date PSDK as it is. Here's the link to the Windows Server 2003 R2 Platform SDK - March 2006 Edition.)
So, first plan of action is to uninstall 3.0 and reinstall the latest 2.x version. Then perhaps we can get started with some actual dev work!