This is one area of .net software that just feels like a step backwards. It's all so 1997. Every single piece of .net software I've got that needs to talk to the internet requires me to set up the proxy.
Every win32 program I use just works.
Now, I can't really blame the developers of these programs (too much). Microsoft didn't make things easy.
Win32 developers used the wininet libraries. These are the libraries that support Internet Explorer, and so have built in support for Internet Explorer's proxy settings, including auto detection and .pac scripts.
.net is not built on top of wininet. Not quite sure why. My best guess is that (amongst other things) most uses of http web requests in .net were expected to be web services, rather than web pages, so wininet's caching was an unnecessary overhead. Whatever the reason, the support for proxy settings isn't exactly as transparent as wininet.
When you created a web request in .net 1.1, the framework would create a proxy for you, based on the values in the config file. You could explicitly define all the properties, or you could use the useSystemDefault attribute to use Internet Explorer's proxy settings.
There were several problems with this approach. IIRC, if the configuration section wasn't in your app.config, you didn't get a proxy - at all (I could be wrong on this, so don't quote me). Secondly, if the IE settings changed, this proxy class (which was cached, by the way) didn't update, so you were stuffed until you restarted your app. Thirdly, automatic proxy settings, based on .pac files, were ignored. And finally, and more importantly for me, no credentials were assigned.
Yep. I'm behind an authenticating proxy. Sucks, doesn't it?
To fix this, you had to explicitly assign credentials to the proxy. If the developer thought about proxies at all, they'd usually forget about this bit. In fact, if the developer did think about proxies, then you were usually in more trouble. They would be more explicit about everything, and leave you nothing to override in the config files. They'd go so far as to expect usernames and passwords to be hard coded in config.
.net 2 improved things a lot. Because I've just spent a good portion of this morning tracing through this looking for a bug I'm going to spell out what happens:
When you create an instance of HttpWebRequest, a default proxy is created for you, based on config. By far your best bet is to accept this, do nothing else and move on.
There's plenty you can do with the config. You can specify everything explicitly, or you can set useSystemDefault=false and get an empty proxy (not too much use. It always treats a url as though it bypasses the proxy, but it does allow you to set credentials). You can even pass in a type that gets spun up via reflection for complete flexibility. Or, you can get a proxy based on IE's settings (useSystemDefault=true).
This last one is quite impressive. It now understands automatically detecting settings and .pac files. It also (finally) responds to changes. And it even respects IE's policies and gets the settings per user or per machine. Another thing to note is that the web proxy is created in the context of the original process, not that of any impersonated users.
The best bit is that it also assigns default credentials to WebProxy.Credentials if you set useDefaultCredentials="true". These default credentials are, effectively, your credentials, and are enough to get you past an authenticating proxy. Hooray!
The slight downside is that this value isn't set by default - I guess for security, you don't want people sending credentials without cause. You can always set it in app.config or machine.config.
That's a lot of flexibility, and all from config. Alternatively, you can set the proxy in code. There are some caveats, so the moral is - don't do this. Don't set a proxy explicitly. Just leave it alone and accept the defaults. They should be flexible enough for the majority of situations.
But if you must set them in code, let's have a look at those caveats. What are your options?
First, the GlobalProxySelection class (originally seen in .net 1.1) allows you to get an empty proxy or returns WebRequest.DefaultWebProxy.
WebRequest.DefaultWebProxy will return either the nice config based class, or a static instance of a proxy class that you set in code - that's right, you can override the config based class. If you do, make sure you get it right.
WebRequest.GetSystemWebProxy() will return the same proxy you'd get if you'd specified useSystemDefault=true in config. However, it short circuits the config file, and it doesn't get credentials. Use this only if you have to (you know, like someone has a gun to your head).
WebProxy.GetDefaultProxy() is deprecated. In .net 1.1 it used to get the IE proxy settings (freshly, each time, which works around the stale settings problem). In .net 2, it's been rerouted to get the proxy settings from IE, but it doesn't touch config and it doesn't get credentials. Please don't use this one.
If you want to set credentials on a proxy, you can specify an explicit set by passing username, domain and password into a new instance of NetworkCredential. Or you can use CredentialCache.DefaultCredentials (which will get you your own credentials). Then set the Credentials property on the WebProxy you've created/retrieved.
A big gotcha is setting WebRequest.UseDefaultCredentials or WebRequest.Credentials. If you've got a proxy assigned to the WebRequest (and by default, you will have) these fields are not used.
A somewhat sneakier gotcha is that the bypass list is processed slightly differently. If you have a bypass list of e.g. *.sticklebackplastic.com, you're saying that you want to bypass the proxy for all subdomains of sticklebackplastic.com. Unfortunately, if you're trying to hit a different port e.g. my.sticklebackplastic.com:8443, this will not match, even though it does in IE. You'll need to add a rule of *.sticklebackplastic.com:8443. This was the big bug that I was chasing today (well, that and the fact that no credentials were set in the config file).
So, there we have it. Short answer is - don't set your own proxy in code and set useDefaultCredentials="true" in the config if you need it.