The presentation briefly covered why we had opted to develop our own infrastructure, how we have gone about implementing all the ancillary services required in a datacentre environment, the network and technology we have built the main services around, how we had gone about detecting and overcoming the inevitable problems which crop up during the day-to-day running of such a facility, and finally the services we are offering at our facility.  Brendan then took questions from the floor relating to the final price tag of the project, as well as addressing some of the technical questions relating to our environmental monitoring network.

For more information download the slides which accompanied the presentation (they’re in PowerPoint format).

• Prepare a kickstart file
• Save the kickstart file in a place accessible for the installer
• Prepare an installation source
• Modify the PXE config so it instructs the kernel to use the kickstart file

1. Kickstart file is a text file which contains a set of instructions/answers for the linux installer so it automatically knows the answers to all the questions normally asked during the installation process (for example, what partition the system should be installed on, what file system is to be used, what are the network settings, what packages are to be installed etc.). You can either use a special tool called Kickstart Configurator (different for every distibution) which will generate a kickstart file for you after answering a set of questions, or create it from scratch yourself. The latter allows you to become familiar with the structure of this file, as well as is the only choice if there is no Kickstart Configurator for your linux distribution. An example kickstart file (which you can use as a template) for a CentOS distribution may look like this:

# System language
lang en_GB
# Language modules to install
langsupport en_GB
# System keyboard
keyboard uk
# System mouse
mouse
# Sytem timezone
timezone Europe/London
# Root password
rootpw root-password
# Reboot after installation
reboot
# Use text mode install
text
# Install OS instead of upgrade
install
# Use Web installation
url --url http://192.168.1.2/distros/centos5.1-iso
# System bootloader configuration
bootloader --location=mbr
# Clear the Master Boot Record
zerombr yes
# Partition clearing information
clearpart --all --initlabel
# Disk partitioning information
part /boot --fstype ext3 --size 100
part / --fstype ext3 --size 1 --grow
part swap --recommended
# System authorization infomation
auth  --useshadow  --enablemd5
# Network information
network --bootproto=dhcp --device=eth1
network --bootproto=dhcp --device=eth0
# Firewall configuration
firewall --disabled
# SELinux configuration
selinux --disabled
# Do not configure XWindows
skipx
# Package install information
%packages --resolvedeps
%post

I am not going to describe each of these options separately as they are all pretty obvious and intuitive. More in detail descriptions as well as the list of all available options can be found here. In short, the above kickstart file will install CentOS from web, without XWindows system, creating 3 ext3 partitions. If you want to use this file for installing other linux distributions you need to take into account that some options may be slightly different. If this is the case you should use an appropriate Kickstart Configurator or find an example kickstart file suitable for your distribution.

2. Once we have a kickstart file we need to make it available for the installer.
There are a few ways to do that. We can put the file on ftp, www or nfs server. We can even store the file directly inside the initrd.img file, so it is accessible for the installer the second the image gets expanded in the memory, although this is the most troublesome way. The most common and easiest way is to put the kickstart file on a www server.

3. Installer has to know where to obtain the packages once the installation process is started. You can choose from a few installation sources: nfs, ftp, http. Again, web installation is the most convenient option. Looking back at our kickstart config you can see that we provided a path to the centos directory:

url --url http://192.168.1.2/distros/centos5.1-iso

The directory is simply the content of a CentOS installation disc. If the only thing you have is an ISO file of the installation disc you can still make use of it by mounting the image on a desired directory. To do so you need to run the following command:

mkdir /var/www/distros/centos5.1-iso
mount -oloop,ro /path/to/centos-iso-file.iso /var/www/distros/centos5.1-iso

After that you should be able to browse the content of the image by entering the directory.

4. The last step is to modify the pxe config. Let us look at the CentOS label that we already have:

LABEL centos5.1
     KERNEL distros/centos5.1/vmlinuz
     APPEND initrd=distros/centos5.1/initrd.img ramdisk_size=6454 ip=dhcp

There are two other options that we need to append to the kernel to make it use our kickstart file. The first one is ‘ks’ which is a path to the kickstart file. In our case it will be:

ks=http://192.168.1.2/distros/kickstart.ks

and the other is ‘ksdevice’ which specifies what network interface should be used in order to retrieve the kickstart file. We may specify interface name like ‘eth0′ or ‘eth1′ however this is not a very handy solution. When you have a server with multiple interfaces you cannot be sure what interface you are actually using. Connecting network cable to the first port on your NIC does not necessarily guarantee that the kernel will recognize this interface as eth0. In such a situation the installer will not be able to retrieve your kickstart file. To avoid this, we can specify ‘bootif’ as the ksdevice. This will make the installer use the interface that we booted from. For this option to work, we also need to add ‘IPAPPEND 2′ option just above the ‘APPEND’ string.

To sum up, the modified label should look like this:

LABEL centos5.1
KERNEL distros/centos5.1/vmlinuz
IPAPPEND 2
APPEND initrd=distros/centos5.1/initrd.img ramdisk_size=6454 ip=dhcp nofb ksdevice=bootif ks=http://192.168.1.2/distros/kickstart.ks

One thing to note here is that the bootif option is not supported by all installers. Therefore if you are having problems with loading your kickstart file, try to specify the exact interface (bearing in mind that the first interface on your NIC does not have to be eth0 interface).

In the past we have operated a fairly re-active strategy to fixing server problems. For example; if we noticed, or someone told us, that a site was running slowly, we would look into it. If a site went down, we would fix it. Of course, in the long run, this isn’t a great way to look after your network. As the sayings go, “a stitch in time saves nine” and “a ounce of prevention is worth a pound of cure”. The same is true with server hosting.

So over the last year, we have begun to implement a much improved strategy to network maintenance that involves pro-actively monitoring all our server resources and identifying potential problems before they occur.

In this blog post, we will describe a real world example of this form of problem spotting that happened to us today.

Zabbix

We now use Zabbix as our main monitoring system. Zabbix monitors indicators such as disk space, memory available, CPU usage, load average and network usage across all our physical machines. Today we spotted a Zabbix graph that looked a little extraordinary.

As you can see from the graph there was a spike in activity on an hourly basis. Look closer and you can see this was happening at 28 minutes past the hour.

From that most server admins will think cronjob! So did we, so we had a look at the server which houses several vServers and on this server we run mirror.fubra.com which is a mirror service we provide to the open source community.

Anyway one of the site’s we mirror is uk3.php.net and as a PHP mirror they install webalizer on the mirror so that the traffic can be tracked. When we set the mirror up it was set-up in a bit of a hurry and we forgot something. Paul describes his chain of thought: “I was thinking to myself, how the @£@$ is that webalizer process causing a constant load avg of 4 for an hour? There are only 6000 visitors per month so it should only take a few seconds!”

Anyway, Mark checked the log files and it turns out they were not being rotated, so we have a 2.7GB log file that webalizer is processing each and every hour over an ATAoE storage network that we have set-up.

This meant this particular problem was causing network load, fileserver load, and web server load all because of a simple error forgetting to rotate a log. As it happens this didn’t bring it to a halt and everything still worked just fine, both before and after we fixed the problem.

To fix things Mark simply installed logrotate, and now it is working a lot better. The result of this is that we can squeeze more value out of our existing hardware.

Becoming more proactive in the monitoring of all our services is a luxury we have been working towards. Up until recently we have been fighting just to get things up and working but now that we have some more server admins and we are looking for even more talented ones we are starting to really get proactive in this area.

We have been quiet for a while on the subject of the mac minis we installed into LINX at Telehouse several months ago…

You may remember the previous article, basically we are using a pair of Mac Mini computers to connect our hosting platform to the LINX Internet exchange in London.

Having a connection to LINX enables us to get much closer to our end users and to our primary content providers, meaning our information moves faster and at lower cost.

Our decision to use the Mac Minis as routers was based on a combination of economic factors. Compared with the typical Cisco 7200 or Juniper M-series routers, the Minis use less space and a fraction of the power whilst providing a massive 2GB of RAM for routing tables, a Core2 DUO 1.83GHz CPU and Gigabit network interface for a tiny fraction of the cost.

Whilst some old-school network purists might have their own opinion on software routers, we have used Linux based routers at our edge and core for over four years now with no issues at all and saved tens of thousands of pounds in routing hardware. So far, packet-for-packet the Minis have performed every bit as well as our 1u rackmount routers in use at our less power constrained sites.

Now that the initial setup is completed, I have been asked to into go more detail about the system and how we solved the various issues encountered along the way. And there have been many issues to solve, mainly because the Mac Mini wasn’t really designed for use as a border router.

So first, the hardware issues…

Hardware problem 1 – Booting without a monitor

The Mac Mini is designed for use as a desktop computer. As such it expects to have a monitor attached – and somewhat unusually will not boot without one.

To get around this problem we made up some svga dongles as described here and connected them using the svga->dvi adapters supplied by Apple.

Granted, from a purist perspective an svga dongle sticking out of the back does spoil the form of the Mini slightly, but in another way it looks like a chromed exhaust pipe, quite fitting for our little hot-hatch routers!

Hardware problem 2 – Automatic power-on

The next hardware problem encountered was how to make the minis boot automatically after a power cut/cycle. Fortunately this was also figured out quite easily, it was simply a case of adding the following line early in the boot process:

  # setpci -s 0:1f.0 0xa4.b=0

Hardware problem 3 – complex routing on a single network interface

The final hardware problem – well not really a problem, but something I often get asked about – is how the networking works. The Mini has just one network card and most people think of a router as having more than one physical network interface.

However, by installing a decent ‘managed’ network switch between our physical network connections and the router, we are able to use VLAN technology (802.1q VLAN Tagging to be precise) to provide multiple secure ‘virtual’ network connections to Linux.

Problem solved. Next, software and driver issues.

We run Ubuntu-Server 7.04 as the operating system on the Minis. For those who aren’t in the know yet, Ubuntu is one of the leading Linux software distributions. Fubra is no stranger to Linux with more than 150 Fedora, Centos and Ubuntu Linux systems deployed across the network as workstations, web servers, email servers, databases and routers.

Software problem 1 – Dodgy ethernet drivers…

Probably the biggest problem we had with the Minis was getting the network cards to remain stable under heavy load. The culprit in this case was what appears to be the buggy (aka “experimental” in kernel speak SKY2 network card driver supplied in the Linux kernel.

In the end we decided to stop using the SKY2 driver altogether until it is more stable. Instead we chose to use the Marvell supplied driver which can be found here. Since switching to the Marvell driver the network card has remained rock solid and we have had no problems or device lockups at all.

Software problem 2 – Connecting to the Internet. Oh yeah, and support for md5 passwords…

For most people, connecting to the Internet means subscribing to a broadband service, which typically provides a single route out to the Internet at between 2 and 20Mb/s. As a hosting company with a lot of high-traffic websites, we have multiple connections to the Internet running at between 100 and 1000Mb/s, enabling us to handle a lot of simultaneous visitors without slowing down.

The other main reason for having multiple connections is that we can survive ISP failures on our network, but implementing this level of protection requires another software protocol running on the router called Border Gateway Protocol, or BGP.

BGP is like a route-planning system for the Internet. In much the same way as an in-car, GPS-based navigation system does, it knows of every route between any two given points on the Internet and can calculate which is the best path to take.

The only issue we had with BGP was getting md5 signatures (a common authentication method for BGP routers) to work – and judging by the Quagga mailing list and wiki this has been a perennial problem for quite some time. I won’t go into too much detail but from what I can tell the resolution was quite typical for an open source project:

1. Some kernel developers suggest md5 signing has no place in the kernel – it should go into userspace.
2. Some userspace developers feel the most appropriate place for md5 signing is in the kernel.
3. Both solutions end up developed… [bgpmd5.pl] [md5qd] [Kernel >=2.6.20]

In the end we went for the following recipe:

• Linux Kernel 2.6.22 (compiled from Vanilla sources)
• Quagga 0.99.9 plus Solinno patch found here

Note it took a couple of extra steps not all that well documented anywhere to get quagga to work properly with the in-kernel support for md5 signing:

First, since the patch was taken against quagga-0.99.8, it has to be applied as follows:

  # tar -zxvf quagga-0.99.9.tar.gz
  # cd quagga-0.99.9
  # patch -p1 <../quagga_md5_bsd_linux_v7.diff

Next, the sources need to be configured with an option to disable IPV6 (note that this disables support for IPV6 in Quagga altogether):

  # ./configure --enable-ipv6=no --localstatedir=/var/run/quagga --sysconfdir=/etc/quagga

To enable the md5 support provided by the Solinno patch, an extra line should be added to config.h before compiling and installing:

  # echo "#define HAVE_TCP_MD5SIG" >>config.h
  # make && make install

I’d like to thank all the guys on the Quagga mailing list who have worked on the md5 problem over the last couple of years.

Having used both of the user-space solutions in production (bgpmd5.pl, md5qd), I can honestly say that the in-kernel method seems to be far more reliable once you get it working and “just feels right”.

Plus a special shout to Dunc from The Bunker who has been working on the same bgp md5 problems for just about as long as I have

So how fast will our little hot-hatches go?

To be fair, I’m not yet 100% sure what we can expect from our Mini routers but the bottom line is, with a 1.8GHz processor, 2GB of ram for holding large routing tables, a Gigabit network interface and a liberal sprinkling of kernel tweaks I fully expect the routers to handle several hundred Megabits per second (and yes, before you shout at me I know – its really about packets per second, not Megabits!).

Hopefully we’ll get the opportunity to fully test some Minis under lab conditions to get some real numbers. When we do, I’ll be sure to post the results in a subsequent article.

I’m convinced that one day we will have to invest in something a little more substantial, after all a hot hatch is great but it doesn’t have enough room for four kids and a pram.

Until then we’ll enjoy the lower fuel consumption and lower purchase cost of our Mac Minis.

The response so far has been overwhelmingly positive which makes me really glad we joined LINX in the first place. At the rate we are going we may have over 100 sessions live tomorrow with a following wind and that would be great progress.

This means our sites should be quicker to load on all the providers who connect to us directly and we will be able to spider all the web sites we are starting to crawl at a lower cost which is great news on both counts.

We still have quite a way to go before we can announce our long awaited hosting service HostOn.it but things are progressing at a swift pace now and everything is looking positive.

In addition to the network engineering we have been doing we have also recently started work on a caching suite that we are going to deploy on all our web sites to reduce the bandwidth and connection overheads they create as well as optimising the HTML to speed load times even further. The suite of tools will optimise all our web sites eliminating the loading of unnecessary objects using user caching and other techniques. In our test environment it has yielded substantially faster load times on our web sites already.

I’m excited about all these developments because we have worked hard to make efficiency and process improvements inside Fubra for the past few years and it is great to see all the hard work which sometimes isn’t as visible as a new feature or web site really paying off.

LINX handles 95% of total UK Internet traffic, and as their newest members we had to come up with a clever solution to keep costs low and speeds high. Utilitising just 3U of rackspace, we were able to install 2 low latency HP gigabit switches and a pair of 1.83 GHz Mac Minis with 2 GB Ram, giving us fully redundant connections to the largest Internet Exchange Point in the world.

Fulfilling our environmental obligations, the total power draw of this setup is less than 2 ordinary household lightbulbs (< 120W).

Our 2 mac mini boxes and HP switches in the LINK rack

Another mac mini router picture

Another mac mini router picture

Our engineers, Nigel Marett and Mark Sutton, who installed the kit said the contrast between our tiny Mac Minis and the existing telco router kit that other people were using was striking.

Nigel commented, “It is kinda a strange sight mate: you walk into that room, and there is a Juniper m120 and another even larger (half rack) router, one of them has an OC48 (STM-15) and three gbit fibres coming out of it, the other one a whole bunch of fibre, and then you get to our rack and there’s two mini’s!”

Background

Recently we have been re-designing our core hosting network and IP connectivity, and as part of this overhaul we decided to join LINX. For those of you who don’t know, an Internet Exchange is where a bunch of ISPs and content providers get together to swap traffic. This saves money and improves network performance as you don’t have to use a 3rd party transit provider to carry data on your behalf.

Since there is a shortage of power in most London data centres, and space is at a premium, our solution would have to work within tight constraints. After searching high and low for suitable dedicated hardware routers, it was clear that Cisco, Juniper and Extreme Networks’ offerings were all too big and power hungry for the job. Someone said “what about a Mac Mini running Quagga?” and the rest, as they say, is history.

Currently our total traffic is at 42Mbps, and we hope that Mac Minis will cope with anything up to several hundred megabits per second on their gigabit interfaces.

Setup

If you want to make your own iRouter setup, you will need the following:

• 2 x HP Procurve 1800 24G Switches
• 2 x 1.83GHz Intel Core Duo Mac Minis (with 2Gb Memory upgrade)
• Ubuntu 7.04 Feisty Fawn (Server Edition) + OpenSSH + Quagga
• 2 x 100Mbps connections to LINX (eXtreme LAN and Foundry LAN) or similar

Fixes

If you plan on using a Mac Mini as a server with Ubuntu 7.04, you need to add the following to the end of your rc.local

setpci -d 8086:27b9 0xa4.b=0

This will fix the power restore status, by telling the EFI not to reset the power flag on reboot. The machine will then auto-power on after a power cut.

As anyone running a lot of web servers knows well the old model of pricing based on space is now mostly out the window and now the major consideration in data centres we look at is power. We are fairly power hungry customer already as you might expect with the site of our network so we have been doing a lot of research and development work to try and reduce our power costs which are growing with us.

Up until recently we had focused on the software and platform solution side of things. By virtulising our hardware demand we have been able to get the best out of our current servers and turn machines that we are not using off and on via our network. This has made a huge difference for us but our hardware when on still sucks a lot of power. We needed a hardware partner that was as obsessed about hardware saving power as we were about optimising our code to be more efficient.

Well we found them, Rackable Systems, and we have just made a huge investment in two new racks of servers. They have lots of advantages over the current batches of 1U servers we are using at the moment but the most exciting part is that they can pack a load more servers into the same space and because the servers run on DC power which is converted from AC only once at the top of the rack and the cooling is very cleverly designed. So we will be saving power and therefore money as well as saving space and therefore more money.

It’s pretty wonderful news because it means we will be able to deliver the services we are working on at a lower cost and so I am very excited about taking delivery of the racks.

I think they may have a new customer for life because the latest product ‘a data centre in a container’, just released called Concentro, is definitely what we are aiming for as we grow.

It also means their customers will get to see our web sites faster and will experience lower latency times when accessing content on our network.

By joining we are contibuting to the IWF (Internet Watch Foundation) and LINX acts as lobbying group to help influence decisions on the direction of the development of Internet technology and policy.

We are pleased that we have been accepted into LINX and look forward to working with other members to ensure both our customers and theirs benefit.

inetnum:        87.124.0.0 - 87.125.255.255
netname:        UK-FUBRA-20050808
descr:          Fubra Limited
country:        GB
org:            ORG-FL12-RIPE
admin-c:        BREN1-RIPE
tech-c:         NIGE-RIPE
status:         ALLOCATED PA
mnt-by:         RIPE-NCC-HM-MNT
mnt-lower:      FUBRA-MNT
mnt-routes:     FUBRA-MNT
mnt-domains:    FUBRA-MNT
source:         RIPE # Filtered

Anyone who lives in those areas and would be interested in signing up to this pilot ADSL2+ service should contact paul@fubra.com.