There’s no delicate way to put this, so I’m just going to have to go ahead and say it; at work, we write automated acceptances tests in a BDD style, using cucumber and Ruby, and using FireWatir to automate Firefox.
Nothing controversial there, it’s just not the most thrilling of opening sentences.
This all started with the need to send a custom HTTP header to a web page we were testing. Now, we could write that using Ruby’s Net::HTTP module, but that would require also writing stuff to manage logins and cookies, and frankly, I’d rather let Firefox handle all that. It just needs to be convinced to send that extra header.
Now, FireWatir has a very interesting implementation. It uses an extension for Firefox called JSSh - a JavaScript Shell (which seems rather tricky to download – try the FireWatir download site). This extension starts listening on a socket, to which you can send raw JavaScript, which JSSh evaluates and returns the output. It exposes a fairly simple API that allows you to e.g. enumerate open windows, navigate to web pages, and access the DOM. FireWatir exercises this quite extensively.
Firefox really likes JavaScript. After all, most extensions are written in JavaScript.
And that right there is the key.
Firefox extensions are written in JavaScript, and they can do *anything* with the browser. So, I just have to figure out how an extension would add a header to a request. One quick Google later, and I’ve found nsIWebNavigation.
Firefox is written to be astonishingly componentised. It is implement on a platform called XPCOM (cross-platform COM, very much like Microsoft’s COM) that exposes a huge amount of functionality. All of which is publicly accessible to those JavaScript extensions.
And nsIWebNavigation is an XPCOM interface all about navigating the web browser, and one method it exposes is loadURI(url, flags, referrer, postData, headers). This method allows me to add custom headers to a navigation request – exactly what I’m looking for. I now have two small problems – how do I get my hands on an instance of nsIWebNavigation, and how do I marshal up the data into the headers parameter?
The first question was answered through a bit of trial and error. If you telnet into JSSh (telnet localhost 9997) you can start investigating your surroundings. You’ve got an interactive shell from which you can run commands, such as getWindow(). But if you type getWindow without the parentheses, you get the JS listing of the function definition. One such example lead me to discover that the higher-level API that JSSh exposes for navigation simply calls into the webNavigation property of the browser variable. And this property is an instance of nsIWebNavigation.
The second problem was solved by Google. I need an instance of nsIInputStream to represent the headers. Turns out I can create an instance of nsIStringInputStream, set the data from a JS string and pass that to loadURI. All of which gives us the following JavaScript:
var headers = Components.classes['@mozilla.org/io/string-input-stream;1']
.createInstance(Components.interfaces.nsIStringInputStream);
headers.setData("X-Forwarded-For: 10.15.142.22\r\n", 31);
#{BROWSER_VAR}.webNavigation.loadURI("http://...", 0, null, null, headers);
The first line looks like some heavy magic. I won’t confess to knowing exactly what’s going on here, but it’s safe to assume that it’s creating an instance of a named XPCOM class and returning the nsIStringInputStream interface implemented by that instance. The second line sets the data into the object, pushing in a string (the loadURI docs state that each header must be separated by a carriage return/line feed pair) and the length of the string. The headers variable is then passed to loadURI, and ta-da! the browser is now navigated to the given URL, and the custom header is sent.
Of course, that’s just the JavaScript. We need to be able to use this from Ruby. I opened up the Firewatir::Firefox class and added:
# Essentially a copy of goto(uri) but can pass headers through the request
# Pass a list of headers, e.g. [ "X-Forwarded-For: 10.15.142.22", "cheese: toast" ]
# (Note that we should also be able to post data through this mozilla method)
def goto_with_headers(url, headers)
#set_defaults()
get_window_number()
set_browser_document()
h = ""
headers.each {|value| h += "#{value}\\r\\n" }
# Load the given url.
jssh_command = "var headers = Components"
jssh_command += ".classes['@mozilla.org/io/string-input-stream;1']"
jssh_command += ".createInstance(Components.interfaces.nsIStringInputStream);"
jssh_command += " headers.setData(\"#{h}\", #{h.length});"
# first null is referrer, second is postData
jssh_command += " #{BROWSER_VAR}.webNavigation.loadURI(\"#{url}\", 0, null,"
jssh_command += " null, headers);"
$jssh_socket.send("#{jssh_command}\n", 0)
read_socket()
wait()
end
Note that the carriage return/line feed escape characters and the quotes in setData have been escaped – we’re writing Ruby that is going to be writing JavaScript. And as the comment says – call it with an array of headers that you’d like to pass to the server.
So there we have it. FireWatir, via JSSh, has a much larger API available to it, thanks to XPCOM. This opens the door to some very interesting possibilities.
I’m aware this is quite an exposition heavy post, so I’ll do an executive summary with a few more examples (would automating Firefox preferences be useful?) and then we’ll get really advanced.