This is taking us quite some time however, as it contains hundreds of gigabytes of production storage. We hope to get this process completed later today.

Platform Improvements

On Tuesday we are expecting to receive a replacement for the failed unit, at which point we will be making some fundamental changes to our storage model to ensure that any future failures of this type can in future be recovered much more quickly. This will be done by replicating all data to a warm standby array that can be switched in immediately in the case of an array going bad.

Although there is still more planning to do, we expect that this measure will massively reduce the impact of any future storage array issues.

In addition to improving our storage array architecture, we are also making other improvements.

The most fundamental change is that we are finally eliminating all of our Lustre-backed storage, and replacing all customer volumes with raw volumes hosted directly on the storage array itself. This has been planned for some time, and will reduce the complexity of our storage to provide much fewer points of failure. The difference will be noticed by all customers previously hosted on our Lustre filesystems.

In addition to stability this will also improve filesystem performance as there will be fewer servers and network round-trips involved in the stack.

Improving Communication

Another area we will concentrate on is communication. We recognise that improvements need to be made when it comes to giving a realistic resolution timescale, and are going to work on this in the coming days to put in place an improved system that enables us to communicate more effectively and respond faster during events such as this.

On this occasion, due to the scale of the outage we had difficulty keeping everyone informed in a timely manner as we scrambled to get things fixed. In the first hours of response it can be very difficult to give an idea of timescales, as it can take some time to get to the root cause. This is much more difficult when the issue lies in storage.

As the week unfolds we will make further progress announcements. In the time being if you experience any problems at all then please contact us via the usual channels and we will rectify the problem as quickly as possible.

I’d like to offer our sincere apologies to all customers affected by these recent issues, and give our assurance that the measures we are taking will dramatically improve our hosting platform for the future.

The presentation briefly covered why we had opted to develop our own infrastructure, how we have gone about implementing all the ancillary services required in a datacentre environment, the network and technology we have built the main services around, how we had gone about detecting and overcoming the inevitable problems which crop up during the day-to-day running of such a facility, and finally the services we are offering at our facility.  Brendan then took questions from the floor relating to the final price tag of the project, as well as addressing some of the technical questions relating to our environmental monitoring network.

For more information download the slides which accompanied the presentation (they’re in PowerPoint format).

Up until now we’ve hosted our own e-mail servers in-house, and whilst that has given us a high degree of flexibility, it’s also been a pain to configure, maintain and keep spam free!

We’re particularly happy with gmail’s filters – previously we had to edit a maildrop filter file on our server in order to set up new rules. Not only does Google provide a simple interface to configure them, it also gives us the ability to choose whether they are marked as read when they come in. I know it’s possible to replicate this in a desktop mail client, but it’s much better to do it on the server – that way if you access your e-mails from multiple devices, they will be organised the same way on each one. This is particularly handy for us as we receive a fair amount of auto-generated notification e-mails from our various web sites which we don’t need to read as they come in but we do like to keep to refer back to.

So the key benefits to us are:

• Simplified and improved filter configuration
• Improved spam detection
• Excellent web mail platform
• Shared calendars
• Easier Administration for setting up new users and groups.
• Takes less time to maintain.

We also plan to become a Google App’s Authorised Reseller over the next few months, so that we can resell these services via hoston.it

• Prepare a kickstart file
• Save the kickstart file in a place accessible for the installer
• Prepare an installation source
• Modify the PXE config so it instructs the kernel to use the kickstart file

1. Kickstart file is a text file which contains a set of instructions/answers for the linux installer so it automatically knows the answers to all the questions normally asked during the installation process (for example, what partition the system should be installed on, what file system is to be used, what are the network settings, what packages are to be installed etc.). You can either use a special tool called Kickstart Configurator (different for every distibution) which will generate a kickstart file for you after answering a set of questions, or create it from scratch yourself. The latter allows you to become familiar with the structure of this file, as well as is the only choice if there is no Kickstart Configurator for your linux distribution. An example kickstart file (which you can use as a template) for a CentOS distribution may look like this:

# System language
lang en_GB
# Language modules to install
langsupport en_GB
# System keyboard
keyboard uk
# System mouse
mouse
# Sytem timezone
timezone Europe/London
# Root password
rootpw root-password
# Reboot after installation
reboot
# Use text mode install
text
# Install OS instead of upgrade
install
# Use Web installation
url --url http://192.168.1.2/distros/centos5.1-iso
# System bootloader configuration
bootloader --location=mbr
# Clear the Master Boot Record
zerombr yes
# Partition clearing information
clearpart --all --initlabel
# Disk partitioning information
part /boot --fstype ext3 --size 100
part / --fstype ext3 --size 1 --grow
part swap --recommended
# System authorization infomation
auth  --useshadow  --enablemd5
# Network information
network --bootproto=dhcp --device=eth1
network --bootproto=dhcp --device=eth0
# Firewall configuration
firewall --disabled
# SELinux configuration
selinux --disabled
# Do not configure XWindows
skipx
# Package install information
%packages --resolvedeps
%post

I am not going to describe each of these options separately as they are all pretty obvious and intuitive. More in detail descriptions as well as the list of all available options can be found here. In short, the above kickstart file will install CentOS from web, without XWindows system, creating 3 ext3 partitions. If you want to use this file for installing other linux distributions you need to take into account that some options may be slightly different. If this is the case you should use an appropriate Kickstart Configurator or find an example kickstart file suitable for your distribution.

2. Once we have a kickstart file we need to make it available for the installer.
There are a few ways to do that. We can put the file on ftp, www or nfs server. We can even store the file directly inside the initrd.img file, so it is accessible for the installer the second the image gets expanded in the memory, although this is the most troublesome way. The most common and easiest way is to put the kickstart file on a www server.

3. Installer has to know where to obtain the packages once the installation process is started. You can choose from a few installation sources: nfs, ftp, http. Again, web installation is the most convenient option. Looking back at our kickstart config you can see that we provided a path to the centos directory:

url --url http://192.168.1.2/distros/centos5.1-iso

The directory is simply the content of a CentOS installation disc. If the only thing you have is an ISO file of the installation disc you can still make use of it by mounting the image on a desired directory. To do so you need to run the following command:

mkdir /var/www/distros/centos5.1-iso
mount -oloop,ro /path/to/centos-iso-file.iso /var/www/distros/centos5.1-iso

After that you should be able to browse the content of the image by entering the directory.

4. The last step is to modify the pxe config. Let us look at the CentOS label that we already have:

LABEL centos5.1
     KERNEL distros/centos5.1/vmlinuz
     APPEND initrd=distros/centos5.1/initrd.img ramdisk_size=6454 ip=dhcp

There are two other options that we need to append to the kernel to make it use our kickstart file. The first one is ‘ks’ which is a path to the kickstart file. In our case it will be:

ks=http://192.168.1.2/distros/kickstart.ks

and the other is ‘ksdevice’ which specifies what network interface should be used in order to retrieve the kickstart file. We may specify interface name like ‘eth0′ or ‘eth1′ however this is not a very handy solution. When you have a server with multiple interfaces you cannot be sure what interface you are actually using. Connecting network cable to the first port on your NIC does not necessarily guarantee that the kernel will recognize this interface as eth0. In such a situation the installer will not be able to retrieve your kickstart file. To avoid this, we can specify ‘bootif’ as the ksdevice. This will make the installer use the interface that we booted from. For this option to work, we also need to add ‘IPAPPEND 2′ option just above the ‘APPEND’ string.

To sum up, the modified label should look like this:

LABEL centos5.1
KERNEL distros/centos5.1/vmlinuz
IPAPPEND 2
APPEND initrd=distros/centos5.1/initrd.img ramdisk_size=6454 ip=dhcp nofb ksdevice=bootif ks=http://192.168.1.2/distros/kickstart.ks

One thing to note here is that the bootif option is not supported by all installers. Therefore if you are having problems with loading your kickstart file, try to specify the exact interface (bearing in mind that the first interface on your NIC does not have to be eth0 interface).

There are some general opinions that tell you to never use .htaccess files on your web server, putting all the options in the main configuration file.

Here is a quote from Apache documentation:

When AllowOverride is set to allow the use of .htaccess files, Apache will look in every directory for .htaccess files. Thus, permitting .htaccess files causes a performance hit, whether or not you actually even use them! Also, the .htaccess file is loaded every time a document is requested.

which sounds pretty scary! Since .htaccess files offer many advantages including much greater flexibility (possibility of making changes on the fly without root access, and without a need to restart the server), we decided not to give up on using them that easily and to carry out some tests ourselves to see the actual impact they have on performance.

Setting up a test environment

First of all we need to edit our Apache httpd.conf file in order to set up two separate directories and enable .htaccess files for one of them:

<Directory /var/www/html/htaccess-enabled>
    Options FollowSymLinks
    AllowOverride All
</Directory>

<Directory /var/www/html/htaccess-disabled>
    AddHandler php5-script .php_script
    Options FollowSymLinks
    AllowOverride None
</Directory>

Now we need to create the directories:

# mkdir /var/www/html/htaccess-enabled
# mkdir /var/www/html/htaccess-disabled

and .htaccess file:

# echo 'AddHandler php5-script .php_script'
>/var/www/html/htaccess-enabled/.htaccess

We also need to create simple pages so we could request them later:

# echo '.htaccess enabled'
>/var/www/html/htaccess-enabled/enabled.html
# echo '.htaccess disabled'
>/var/www/html/htaccess-disabled/disabled.html

After restarting the server:

# apachectl restart

we can start our tests.

Testing

By using the following command we can see how long it will take to request each of the pages 500 times:

# time for i in `seq 5000`; do
curl http://localhost/htaccess-disabled/disabled.html >/dev/null;done
real    0m51.677s
user    0m28.482s
sys     0m21.101s

# time for i in `seq 5000`; do
curl http://localhost/htaccess-enabled/enabled.html >/dev/null;done
real    0m52.217s
user    0m28.206s
sys     0m21.317s

In order to get more comprehensive results we can use ab (Apache Benchmark) program as follows:

# ab -n 5000 -c10 http://localhost/htaccess-disabled/disabled.html
# ab -n 5000 -c10 http://localhost/htaccess-enabled/enabled.html

-n denotes number of requests, and -c number of requests to be sent at the same time.

Repeating the test 5 times for each of the URL’s we achieve the following results:

That gives us a difference of around 6.6% less requests per second while .htaccess is turned on.

We have been quiet for a while on the subject of the mac minis we installed into LINX at Telehouse several months ago…

You may remember the previous article, basically we are using a pair of Mac Mini computers to connect our hosting platform to the LINX Internet exchange in London.

Having a connection to LINX enables us to get much closer to our end users and to our primary content providers, meaning our information moves faster and at lower cost.

Our decision to use the Mac Minis as routers was based on a combination of economic factors. Compared with the typical Cisco 7200 or Juniper M-series routers, the Minis use less space and a fraction of the power whilst providing a massive 2GB of RAM for routing tables, a Core2 DUO 1.83GHz CPU and Gigabit network interface for a tiny fraction of the cost.

Whilst some old-school network purists might have their own opinion on software routers, we have used Linux based routers at our edge and core for over four years now with no issues at all and saved tens of thousands of pounds in routing hardware. So far, packet-for-packet the Minis have performed every bit as well as our 1u rackmount routers in use at our less power constrained sites.

Now that the initial setup is completed, I have been asked to into go more detail about the system and how we solved the various issues encountered along the way. And there have been many issues to solve, mainly because the Mac Mini wasn’t really designed for use as a border router.

So first, the hardware issues…

Hardware problem 1 – Booting without a monitor

The Mac Mini is designed for use as a desktop computer. As such it expects to have a monitor attached – and somewhat unusually will not boot without one.

To get around this problem we made up some svga dongles as described here and connected them using the svga->dvi adapters supplied by Apple.

Granted, from a purist perspective an svga dongle sticking out of the back does spoil the form of the Mini slightly, but in another way it looks like a chromed exhaust pipe, quite fitting for our little hot-hatch routers!

Hardware problem 2 – Automatic power-on

The next hardware problem encountered was how to make the minis boot automatically after a power cut/cycle. Fortunately this was also figured out quite easily, it was simply a case of adding the following line early in the boot process:

  # setpci -s 0:1f.0 0xa4.b=0

Hardware problem 3 – complex routing on a single network interface

The final hardware problem – well not really a problem, but something I often get asked about – is how the networking works. The Mini has just one network card and most people think of a router as having more than one physical network interface.

However, by installing a decent ‘managed’ network switch between our physical network connections and the router, we are able to use VLAN technology (802.1q VLAN Tagging to be precise) to provide multiple secure ‘virtual’ network connections to Linux.

Problem solved. Next, software and driver issues.

We run Ubuntu-Server 7.04 as the operating system on the Minis. For those who aren’t in the know yet, Ubuntu is one of the leading Linux software distributions. Fubra is no stranger to Linux with more than 150 Fedora, Centos and Ubuntu Linux systems deployed across the network as workstations, web servers, email servers, databases and routers.

Software problem 1 – Dodgy ethernet drivers…

Probably the biggest problem we had with the Minis was getting the network cards to remain stable under heavy load. The culprit in this case was what appears to be the buggy (aka “experimental” in kernel speak SKY2 network card driver supplied in the Linux kernel.

In the end we decided to stop using the SKY2 driver altogether until it is more stable. Instead we chose to use the Marvell supplied driver which can be found here. Since switching to the Marvell driver the network card has remained rock solid and we have had no problems or device lockups at all.

Software problem 2 – Connecting to the Internet. Oh yeah, and support for md5 passwords…

For most people, connecting to the Internet means subscribing to a broadband service, which typically provides a single route out to the Internet at between 2 and 20Mb/s. As a hosting company with a lot of high-traffic websites, we have multiple connections to the Internet running at between 100 and 1000Mb/s, enabling us to handle a lot of simultaneous visitors without slowing down.

The other main reason for having multiple connections is that we can survive ISP failures on our network, but implementing this level of protection requires another software protocol running on the router called Border Gateway Protocol, or BGP.

BGP is like a route-planning system for the Internet. In much the same way as an in-car, GPS-based navigation system does, it knows of every route between any two given points on the Internet and can calculate which is the best path to take.

The only issue we had with BGP was getting md5 signatures (a common authentication method for BGP routers) to work – and judging by the Quagga mailing list and wiki this has been a perennial problem for quite some time. I won’t go into too much detail but from what I can tell the resolution was quite typical for an open source project:

1. Some kernel developers suggest md5 signing has no place in the kernel – it should go into userspace.
2. Some userspace developers feel the most appropriate place for md5 signing is in the kernel.
3. Both solutions end up developed… [bgpmd5.pl] [md5qd] [Kernel >=2.6.20]

In the end we went for the following recipe:

• Linux Kernel 2.6.22 (compiled from Vanilla sources)
• Quagga 0.99.9 plus Solinno patch found here

Note it took a couple of extra steps not all that well documented anywhere to get quagga to work properly with the in-kernel support for md5 signing:

First, since the patch was taken against quagga-0.99.8, it has to be applied as follows:

  # tar -zxvf quagga-0.99.9.tar.gz
  # cd quagga-0.99.9
  # patch -p1 <../quagga_md5_bsd_linux_v7.diff

Next, the sources need to be configured with an option to disable IPV6 (note that this disables support for IPV6 in Quagga altogether):

  # ./configure --enable-ipv6=no --localstatedir=/var/run/quagga --sysconfdir=/etc/quagga

To enable the md5 support provided by the Solinno patch, an extra line should be added to config.h before compiling and installing:

  # echo "#define HAVE_TCP_MD5SIG" >>config.h
  # make && make install

I’d like to thank all the guys on the Quagga mailing list who have worked on the md5 problem over the last couple of years.

Having used both of the user-space solutions in production (bgpmd5.pl, md5qd), I can honestly say that the in-kernel method seems to be far more reliable once you get it working and “just feels right”.

Plus a special shout to Dunc from The Bunker who has been working on the same bgp md5 problems for just about as long as I have

So how fast will our little hot-hatches go?

To be fair, I’m not yet 100% sure what we can expect from our Mini routers but the bottom line is, with a 1.8GHz processor, 2GB of ram for holding large routing tables, a Gigabit network interface and a liberal sprinkling of kernel tweaks I fully expect the routers to handle several hundred Megabits per second (and yes, before you shout at me I know – its really about packets per second, not Megabits!).

Hopefully we’ll get the opportunity to fully test some Minis under lab conditions to get some real numbers. When we do, I’ll be sure to post the results in a subsequent article.

I’m convinced that one day we will have to invest in something a little more substantial, after all a hot hatch is great but it doesn’t have enough room for four kids and a pram.

Until then we’ll enjoy the lower fuel consumption and lower purchase cost of our Mac Minis.

The Bunker is an ex-cold war nuclear bunker that has been converted into a state of the art co-location facility. It’s also one of the three data centres we currently use along with IXEurope Heathrow, and TeleHouse North.

The actual data centre floor is situated 30 metres below ground level, and so the first challenge was to get rack down there by lowering it on a crane lift. Since the rack was so heavy, they had to remove all 38 dual xeon servers and take them down seperately.

Mark was impressed with the overall ease of assembly of the Rackable rack, saying “Stripping down, rebuilding and cabling a rack of 38 servers would normally take at least a full day, if not more. We managed to get from loading bay to power-up in less than 6 hours, including a complete network rewire of the system, 126 patch cables in total.”

LINX handles 95% of total UK Internet traffic, and as their newest members we had to come up with a clever solution to keep costs low and speeds high. Utilitising just 3U of rackspace, we were able to install 2 low latency HP gigabit switches and a pair of 1.83 GHz Mac Minis with 2 GB Ram, giving us fully redundant connections to the largest Internet Exchange Point in the world.

Fulfilling our environmental obligations, the total power draw of this setup is less than 2 ordinary household lightbulbs (< 120W).

Our 2 mac mini boxes and HP switches in the LINK rack

Another mac mini router picture

Another mac mini router picture

Our engineers, Nigel Marett and Mark Sutton, who installed the kit said the contrast between our tiny Mac Minis and the existing telco router kit that other people were using was striking.

Nigel commented, “It is kinda a strange sight mate: you walk into that room, and there is a Juniper m120 and another even larger (half rack) router, one of them has an OC48 (STM-15) and three gbit fibres coming out of it, the other one a whole bunch of fibre, and then you get to our rack and there’s two mini’s!”

Background

Recently we have been re-designing our core hosting network and IP connectivity, and as part of this overhaul we decided to join LINX. For those of you who don’t know, an Internet Exchange is where a bunch of ISPs and content providers get together to swap traffic. This saves money and improves network performance as you don’t have to use a 3rd party transit provider to carry data on your behalf.

Since there is a shortage of power in most London data centres, and space is at a premium, our solution would have to work within tight constraints. After searching high and low for suitable dedicated hardware routers, it was clear that Cisco, Juniper and Extreme Networks’ offerings were all too big and power hungry for the job. Someone said “what about a Mac Mini running Quagga?” and the rest, as they say, is history.

Currently our total traffic is at 42Mbps, and we hope that Mac Minis will cope with anything up to several hundred megabits per second on their gigabit interfaces.

Setup

If you want to make your own iRouter setup, you will need the following:

• 2 x HP Procurve 1800 24G Switches
• 2 x 1.83GHz Intel Core Duo Mac Minis (with 2Gb Memory upgrade)
• Ubuntu 7.04 Feisty Fawn (Server Edition) + OpenSSH + Quagga
• 2 x 100Mbps connections to LINX (eXtreme LAN and Foundry LAN) or similar

Fixes

If you plan on using a Mac Mini as a server with Ubuntu 7.04, you need to add the following to the end of your rc.local

setpci -d 8086:27b9 0xa4.b=0

This will fix the power restore status, by telling the EFI not to reset the power flag on reboot. The machine will then auto-power on after a power cut.

As anyone running a lot of web servers knows well the old model of pricing based on space is now mostly out the window and now the major consideration in data centres we look at is power. We are fairly power hungry customer already as you might expect with the site of our network so we have been doing a lot of research and development work to try and reduce our power costs which are growing with us.

Up until recently we had focused on the software and platform solution side of things. By virtulising our hardware demand we have been able to get the best out of our current servers and turn machines that we are not using off and on via our network. This has made a huge difference for us but our hardware when on still sucks a lot of power. We needed a hardware partner that was as obsessed about hardware saving power as we were about optimising our code to be more efficient.

Well we found them, Rackable Systems, and we have just made a huge investment in two new racks of servers. They have lots of advantages over the current batches of 1U servers we are using at the moment but the most exciting part is that they can pack a load more servers into the same space and because the servers run on DC power which is converted from AC only once at the top of the rack and the cooling is very cleverly designed. So we will be saving power and therefore money as well as saving space and therefore more money.

It’s pretty wonderful news because it means we will be able to deliver the services we are working on at a lower cost and so I am very excited about taking delivery of the racks.

I think they may have a new customer for life because the latest product ‘a data centre in a container’, just released called Concentro, is definitely what we are aiming for as we grow.

When Paul showed me WinSCP as a great alternative to FTP I was well chuffed mainly because we haven’t really been able to get to our servers via FTP as hardly any run FTP servers but also because I can use my SSH key to authenticate rather than remembering passwords which I almost never use because they are so hard to remember.

The people behind this project should be proud it is definitely one of my favorite apps.