We have been quiet for a while on the subject of the mac minis we installed into LINX at Telehouse several months ago…

You may remember the previous article, basically we are using a pair of Mac Mini computers to connect our hosting platform to the LINX Internet exchange in London.

Having a connection to LINX enables us to get much closer to our end users and to our primary content providers, meaning our information moves faster and at lower cost.

Our decision to use the Mac Minis as routers was based on a combination of economic factors. Compared with the typical Cisco 7200 or Juniper M-series routers, the Minis use less space and a fraction of the power whilst providing a massive 2GB of RAM for routing tables, a Core2 DUO 1.83GHz CPU and Gigabit network interface for a tiny fraction of the cost.

Whilst some old-school network purists might have their own opinion on software routers, we have used Linux based routers at our edge and core for over four years now with no issues at all and saved tens of thousands of pounds in routing hardware. So far, packet-for-packet the Minis have performed every bit as well as our 1u rackmount routers in use at our less power constrained sites.

Now that the initial setup is completed, I have been asked to into go more detail about the system and how we solved the various issues encountered along the way. And there have been many issues to solve, mainly because the Mac Mini wasn’t really designed for use as a border router.

So first, the hardware issues…

Hardware problem 1 – Booting without a monitor

The Mac Mini is designed for use as a desktop computer. As such it expects to have a monitor attached – and somewhat unusually will not boot without one.

To get around this problem we made up some svga dongles as described here and connected them using the svga->dvi adapters supplied by Apple.

Granted, from a purist perspective an svga dongle sticking out of the back does spoil the form of the Mini slightly, but in another way it looks like a chromed exhaust pipe, quite fitting for our little hot-hatch routers!

Hardware problem 2 – Automatic power-on

The next hardware problem encountered was how to make the minis boot automatically after a power cut/cycle. Fortunately this was also figured out quite easily, it was simply a case of adding the following line early in the boot process:

  # setpci -s 0:1f.0 0xa4.b=0

Hardware problem 3 – complex routing on a single network interface

The final hardware problem – well not really a problem, but something I often get asked about – is how the networking works. The Mini has just one network card and most people think of a router as having more than one physical network interface.

However, by installing a decent ‘managed’ network switch between our physical network connections and the router, we are able to use VLAN technology (802.1q VLAN Tagging to be precise) to provide multiple secure ‘virtual’ network connections to Linux.

Problem solved. Next, software and driver issues.

We run Ubuntu-Server 7.04 as the operating system on the Minis. For those who aren’t in the know yet, Ubuntu is one of the leading Linux software distributions. Fubra is no stranger to Linux with more than 150 Fedora, Centos and Ubuntu Linux systems deployed across the network as workstations, web servers, email servers, databases and routers.

Software problem 1 – Dodgy ethernet drivers…

Probably the biggest problem we had with the Minis was getting the network cards to remain stable under heavy load. The culprit in this case was what appears to be the buggy (aka “experimental” in kernel speak SKY2 network card driver supplied in the Linux kernel.

In the end we decided to stop using the SKY2 driver altogether until it is more stable. Instead we chose to use the Marvell supplied driver which can be found here. Since switching to the Marvell driver the network card has remained rock solid and we have had no problems or device lockups at all.

Software problem 2 – Connecting to the Internet. Oh yeah, and support for md5 passwords…

For most people, connecting to the Internet means subscribing to a broadband service, which typically provides a single route out to the Internet at between 2 and 20Mb/s. As a hosting company with a lot of high-traffic websites, we have multiple connections to the Internet running at between 100 and 1000Mb/s, enabling us to handle a lot of simultaneous visitors without slowing down.

The other main reason for having multiple connections is that we can survive ISP failures on our network, but implementing this level of protection requires another software protocol running on the router called Border Gateway Protocol, or BGP.

BGP is like a route-planning system for the Internet. In much the same way as an in-car, GPS-based navigation system does, it knows of every route between any two given points on the Internet and can calculate which is the best path to take.

The only issue we had with BGP was getting md5 signatures (a common authentication method for BGP routers) to work – and judging by the Quagga mailing list and wiki this has been a perennial problem for quite some time. I won’t go into too much detail but from what I can tell the resolution was quite typical for an open source project:

1. Some kernel developers suggest md5 signing has no place in the kernel – it should go into userspace.
2. Some userspace developers feel the most appropriate place for md5 signing is in the kernel.
3. Both solutions end up developed… [bgpmd5.pl] [md5qd] [Kernel >=2.6.20]

In the end we went for the following recipe:

• Linux Kernel 2.6.22 (compiled from Vanilla sources)
• Quagga 0.99.9 plus Solinno patch found here

Note it took a couple of extra steps not all that well documented anywhere to get quagga to work properly with the in-kernel support for md5 signing:

First, since the patch was taken against quagga-0.99.8, it has to be applied as follows:

  # tar -zxvf quagga-0.99.9.tar.gz
  # cd quagga-0.99.9
  # patch -p1 <../quagga_md5_bsd_linux_v7.diff

Next, the sources need to be configured with an option to disable IPV6 (note that this disables support for IPV6 in Quagga altogether):

  # ./configure --enable-ipv6=no --localstatedir=/var/run/quagga --sysconfdir=/etc/quagga

To enable the md5 support provided by the Solinno patch, an extra line should be added to config.h before compiling and installing:

  # echo "#define HAVE_TCP_MD5SIG" >>config.h
  # make && make install

I’d like to thank all the guys on the Quagga mailing list who have worked on the md5 problem over the last couple of years.

Having used both of the user-space solutions in production (bgpmd5.pl, md5qd), I can honestly say that the in-kernel method seems to be far more reliable once you get it working and “just feels right”.

Plus a special shout to Dunc from The Bunker who has been working on the same bgp md5 problems for just about as long as I have

So how fast will our little hot-hatches go?

To be fair, I’m not yet 100% sure what we can expect from our Mini routers but the bottom line is, with a 1.8GHz processor, 2GB of ram for holding large routing tables, a Gigabit network interface and a liberal sprinkling of kernel tweaks I fully expect the routers to handle several hundred Megabits per second (and yes, before you shout at me I know – its really about packets per second, not Megabits!).

Hopefully we’ll get the opportunity to fully test some Minis under lab conditions to get some real numbers. When we do, I’ll be sure to post the results in a subsequent article.

I’m convinced that one day we will have to invest in something a little more substantial, after all a hot hatch is great but it doesn’t have enough room for four kids and a pram.

Until then we’ll enjoy the lower fuel consumption and lower purchase cost of our Mac Minis.

LINX handles 95% of total UK Internet traffic, and as their newest members we had to come up with a clever solution to keep costs low and speeds high. Utilitising just 3U of rackspace, we were able to install 2 low latency HP gigabit switches and a pair of 1.83 GHz Mac Minis with 2 GB Ram, giving us fully redundant connections to the largest Internet Exchange Point in the world.

Fulfilling our environmental obligations, the total power draw of this setup is less than 2 ordinary household lightbulbs (< 120W).

Our 2 mac mini boxes and HP switches in the LINK rack

Another mac mini router picture

Another mac mini router picture

Our engineers, Nigel Marett and Mark Sutton, who installed the kit said the contrast between our tiny Mac Minis and the existing telco router kit that other people were using was striking.

Nigel commented, “It is kinda a strange sight mate: you walk into that room, and there is a Juniper m120 and another even larger (half rack) router, one of them has an OC48 (STM-15) and three gbit fibres coming out of it, the other one a whole bunch of fibre, and then you get to our rack and there’s two mini’s!”

Background

Recently we have been re-designing our core hosting network and IP connectivity, and as part of this overhaul we decided to join LINX. For those of you who don’t know, an Internet Exchange is where a bunch of ISPs and content providers get together to swap traffic. This saves money and improves network performance as you don’t have to use a 3rd party transit provider to carry data on your behalf.

Since there is a shortage of power in most London data centres, and space is at a premium, our solution would have to work within tight constraints. After searching high and low for suitable dedicated hardware routers, it was clear that Cisco, Juniper and Extreme Networks’ offerings were all too big and power hungry for the job. Someone said “what about a Mac Mini running Quagga?” and the rest, as they say, is history.

Currently our total traffic is at 42Mbps, and we hope that Mac Minis will cope with anything up to several hundred megabits per second on their gigabit interfaces.

Setup

If you want to make your own iRouter setup, you will need the following:

• 2 x HP Procurve 1800 24G Switches
• 2 x 1.83GHz Intel Core Duo Mac Minis (with 2Gb Memory upgrade)
• Ubuntu 7.04 Feisty Fawn (Server Edition) + OpenSSH + Quagga
• 2 x 100Mbps connections to LINX (eXtreme LAN and Foundry LAN) or similar

Fixes

If you plan on using a Mac Mini as a server with Ubuntu 7.04, you need to add the following to the end of your rc.local

setpci -d 8086:27b9 0xa4.b=0

This will fix the power restore status, by telling the EFI not to reset the power flag on reboot. The machine will then auto-power on after a power cut.

It was acquired following the successful acquisition of HousePriceCrash.co.uk last year another forum web site which discusses the possibility of house prices falling.

As a business traditionally Fubra’s growth has come purely organically. But we have spent more than a year working hard on perfecting our internal systems and processes as well as our hosting platform and advertising systems. Now we are ready to step up our growth levels both organically and via acquisition. We are excited about the prospects for both methods of growth.

The purchase of HousePriceCrash.co.uk has given us confidence, that we can successfully integrate a community web site into our network of web sites and make improvements to it. Those improvements have pleased the community and enhanced our traffic and earnings growth from the web site.

As Fubra continues to grow we will look for more web sites to integrate into our portfolio that are priced at sensible earnings multiples. The sites will need to present the opportunity to better monitise the existing traffic they have and also to use our other internal talent and platforms to grow the current traffic.

InsanelyMac.com is a great addtion with over 40,000 visitor per day and 14% of those UK based visits and 60% US based. It adds substantially to our portfolio of sites and is a rare find. Anyone wishing to target this technology savvy audience should contact Brendan on our team page about the options available.