We think this could be a world's first: Forget Cisco or Juniper, yesterday we hooked up two Mac Mini boxes running Quagga in to the London INternet eXchange (LINX) to act as BGP border routers for the Fubra Network.

LINX handles 95% of total UK Internet traffic, and as their newest members we had to come up with a clever solution to keep costs low and speeds high. Utilitising just 3U of rackspace, we were able to install 2 low latency HP gigabit switches and a pair of 1.83 GHz Mac Minis with 2 GB Ram, giving us fully redundant connections to the largest Internet Exchange Point in the world.

Fulfilling our environmental obligations, the total power draw of this setup is less than 2 ordinary household lightbulbs (< 120W).







Our engineers, Nigel Marett and Mark Sutton, who installed the kit said the contrast between our tiny Mac Minis and the existing telco router kit that other people were using was striking.

Nigel commented, "It is kinda a strange sight mate: you walk into that room, and there is a Juniper m120 and another even larger (half rack) router, one of them has an OC48 (STM-15) and three gbit fibres coming out of it, the other one a whole bunch of fibre, and then you get to our rack and there's two mini's!"

Background

Recently we have been re-designing our core hosting network and IP connectivity, and as part of this overhaul we decided to join LINX. For those of you who don't know, an Internet Exchange is where a bunch of ISPs and content providers get together to swap traffic. This saves money and improves network performance as you don't have to use a 3rd party transit provider to carry data on your behalf.

Since there is a shortage of power in most London data centres, and space is at a premium, our solution would have to work within tight constraints. After searching high and low for suitable dedicated hardware routers, it was clear that Cisco, Juniper and Extreme Networks' offerings were all too big and power hungry for the job. Someone said "what about a Mac Mini running Quagga?" and the rest, as they say, is history.

Currently our total traffic is at 42Mbps, and we hope that Mac Minis will cope with anything up to several hundred megabits per second on their gigabit interfaces.

Setup

If you want to make your own iRouter setup, you will need the following:

• 2 x HP Procurve 1800 24G Switches
• 2 x 1.83GHz Intel Core Duo Mac Minis (with 2Gb Memory upgrade)
• Ubuntu 7.04 Feisty Fawn (Server Edition) + OpenSSH + Quagga
• 2 x 100Mbps connections to LINX (eXtreme LAN and Foundry LAN) or similar

Fixes

If you plan on using a Mac Mini as a server with Ubuntu 7.04, you need to add the following to the end of your rc.local

setpci -d 8086:27b9 0xa4.b=0

This will fix the power restore status, by telling the EFI not to reset the power flag on reboot. The machine will then auto-power on after a power cut.

Things are getting a bit busy now on the Fubra network which was part of the reason for our LINX membership that we announced last week. With all the traffic we are getting it will be great to pay less to get our data onto the ISP's networks our users come from but it left us thinking what about the extra servers we are going to need.

As anyone running a lot of web servers knows well the old model of pricing based on space is now mostly out the window and now the major consideration in data centres we look at is power. We are fairly power hungry customer already as you might expect with the site of our network so we have been doing a lot of research and development work to try and reduce our power costs which are growing with us.

Up until recently we had focused on the software and platform solution side of things. By virtulising our hardware demand we have been able to get the best out of our current servers and turn machines that we are not using off and on via our network. This has made a huge difference for us but our hardware when on still sucks a lot of power. We needed a hardware partner that was as obsessed about hardware saving power as we were about optimising our code to be more efficient.

Well we found them, Rackable Systems, and we have just made a huge investment in two new racks of servers. They have lots of advantages over the current batches of 1U servers we are using at the moment but the most exciting part is that they can pack a load more servers into the same space and because the servers run on DC power which is converted from AC only once at the top of the rack and the cooling is very cleverly designed. So we will be saving power and therefore money as well as saving space and therefore more money.

It's pretty wonderful news because it means we will be able to deliver the services we are working on at a lower cost and so I am very excited about taking delivery of the racks.

I think they may have a new customer for life because the latest product 'a data centre in a container', just released called Concentro, is definitely what we are aiming for as we grow.