Best Open source proxy server
I'd certainly recommend Squid on Linux/POSIX/Unix, however where I've used it previously I found it would slow down after running for a couple of weeks and needed to be restarted - but this was simple to schedule. Squid is available as a binary for Microsoft NT platforms - can't comment on how stable it is there.
I've always wondered that whilst proxy server solution works okay, there might be another way to achieve this
Yes - this is a fairly straightforward task. I would expect most routers to be capable of choosing a route based on the type of traffic. Again my experience is predominantly with Linux - where it's quite possible using iptables. Having said that, you do get the benefit of a shared cache with most proxies (including squid). Note that it is not possible to cache SSL traffic on a proxy.
While Apache's mod_proxy is also capable of providing the functionality you describe, I find squid easier to work with.
While there other other open-source proxy tools out there, unless you have a very specific requirement (e.g. SOCKs support, or offline browsing) there's not much point in looking any further than squid. There are also a huge number of add-ons to squid to support all sorts of things like content-filtering, Anti-Virus, authentication.