Thinking Differently... An update on our Mac Mini Routers at LINX

We have been quiet for a while on the subject of the mac minis we installed into LINX at Telehouse several months ago...

You may remember the previous article, basically we are using a pair of Mac Mini computers to connect our hosting platform to the LINX Internet exchange in London.


Having a connection to LINX enables us to get much closer to our end users and to our primary content providers, meaning our information moves faster and at lower cost.

Our decision to use the Mac Minis as routers was based on a combination of economic factors. Compared with the typical Cisco 7200 or Juniper M-series routers, the Minis use less space and a fraction of the power whilst providing a massive 2GB of RAM for routing tables, a Core2 DUO 1.83GHz CPU and Gigabit network interface for a tiny fraction of the cost.

Whilst some old-school network purists might have their own opinion on software routers, we have used Linux based routers at our edge and core for over four years now with no issues at all and saved tens of thousands of pounds in routing hardware. So far, packet-for-packet the Minis have performed every bit as well as our 1u rackmount routers in use at our less power constrained sites.

Now that the initial setup is completed, I have been asked to into go more detail about the system and how we solved the various issues encountered along the way. And there have been many issues to solve, mainly because the Mac Mini wasn't really designed for use as a border router.

So first, the hardware issues...

Hardware problem 1 - Booting without a monitor

The Mac Mini is designed for use as a desktop computer. As such it expects to have a monitor attached - and somewhat unusually will not boot without one.

To get around this problem we made up some svga dongles as described here and connected them using the svga->dvi adapters supplied by Apple.



Granted, from a purist perspective an svga dongle sticking out of the back does spoil the form of the Mini slightly, but in another way it looks like a chromed exhaust pipe, quite fitting for our little hot-hatch routers!

Hardware problem 2 - Automatic power-on

The next hardware problem encountered was how to make the minis boot automatically after a power cut/cycle. Fortunately this was also figured out quite easily, it was simply a case of adding the following line early in the boot process:

  # setpci -s 0:1f.0 0xa4.b=0

Hardware problem 3 - complex routing on a single network interface

The final hardware problem - well not really a problem, but something I often get asked about - is how the networking works. The Mini has just one network card and most people think of a router as having more than one physical network interface.

However, by installing a decent 'managed' network switch between our physical network connections and the router, we are able to use VLAN technology (802.1q VLAN Tagging to be precise) to provide multiple secure 'virtual' network connections to Linux.


Problem solved. Next, software and driver issues.

We run Ubuntu-Server 7.04 as the operating system on the Minis. For those who aren't in the know yet, Ubuntu is one of the leading Linux software distributions. Fubra is no stranger to Linux with more than 150 Fedora, Centos and Ubuntu Linux systems deployed across the network as workstations, web servers, email servers, databases and routers.

Software problem 1 - Dodgy ethernet drivers...

Probably the biggest problem we had with the Minis was getting the network cards to remain stable under heavy load. The culprit in this case was what appears to be the buggy (aka "experimental" in kernel speak ;) SKY2 network card driver supplied in the Linux kernel.

In the end we decided to stop using the SKY2 driver altogether until it is more stable. Instead we chose to use the Marvell supplied driver which can be found here. Since switching to the Marvell driver the network card has remained rock solid and we have had no problems or device lockups at all.

Software problem 2 - Connecting to the Internet. Oh yeah, and support for md5 passwords...

For most people, connecting to the Internet means subscribing to a broadband service, which typically provides a single route out to the Internet at between 2 and 20Mb/s. As a hosting company with a lot of high-traffic websites, we have multiple connections to the Internet running at between 100 and 1000Mb/s, enabling us to handle a lot of simultaneous visitors without slowing down.

The other main reason for having multiple connections is that we can survive ISP failures on our network, but implementing this level of protection requires another software protocol running on the router called Border Gateway Protocol, or BGP.

BGP is like a route-planning system for the Internet. In much the same way as an in-car, GPS-based navigation system does, it knows of every route between any two given points on the Internet and can calculate which is the best path to take.

The only issue we had with BGP was getting md5 signatures (a common authentication method for BGP routers) to work - and judging by the Quagga mailing list and wiki this has been a perennial problem for quite some time. I won't go into too much detail but from what I can tell the resolution was quite typical for an open source project:

1. Some kernel developers suggest md5 signing has no place in the kernel - it should go into userspace.
2. Some userspace developers feel the most appropriate place for md5 signing is in the kernel.
3. Both solutions end up developed... [bgpmd5.pl] [md5qd] [Kernel >=2.6.20]

In the end we went for the following recipe:

• Linux Kernel 2.6.22 (compiled from Vanilla sources)
• Quagga 0.99.9 plus Solinno patch found here

Note it took a couple of extra steps not all that well documented anywhere to get quagga to work properly with the in-kernel support for md5 signing:

First, since the patch was taken against quagga-0.99.8, it has to be applied as follows:

  # tar -zxvf quagga-0.99.9.tar.gz
  # cd quagga-0.99.9
  # patch -p1 <../quagga_md5_bsd_linux_v7.diff

Next, the sources need to be configured with an option to disable IPV6 (note that this disables support for IPV6 in Quagga altogether):

  # ./configure --enable-ipv6=no --localstatedir=/var/run/quagga --sysconfdir=/etc/quagga

To enable the md5 support provided by the Solinno patch, an extra line should be added to config.h before compiling and installing:

  # echo "#define HAVE_TCP_MD5SIG" >>config.h
  # make && make install

I'd like to thank all the guys on the Quagga mailing list who have worked on the md5 problem over the last couple of years.

Having used both of the user-space solutions in production (bgpmd5.pl, md5qd), I can honestly say that the in-kernel method seems to be far more reliable once you get it working and "just feels right".

Plus a special shout to Dunc from The Bunker who has been working on the same bgp md5 problems for just about as long as I have :-)

So how fast will our little hot-hatches go?

To be fair, I'm not yet 100% sure what we can expect from our Mini routers but the bottom line is, with a 1.8GHz processor, 2GB of ram for holding large routing tables, a Gigabit network interface and a liberal sprinkling of kernel tweaks I fully expect the routers to handle several hundred Megabits per second (and yes, before you shout at me I know - its really about packets per second, not Megabits!).

Hopefully we'll get the opportunity to fully test some Minis under lab conditions to get some real numbers. When we do, I'll be sure to post the results in a subsequent article.

I'm convinced that one day we will have to invest in something a little more substantial, after all a hot hatch is great but it doesn't have enough room for four kids and a pram.

Until then we'll enjoy the lower fuel consumption and lower purchase cost of our Mac Minis.

--
complexmind
aka Mark Sutton - contact me at mark at fubra dot com