We offer several partnership options to sellers, but the one I’ll talk about here is where we sell their home furnishings directly from furnish.co.uk, just like a marketplace. For example, a customer finds a coffee table on furnish.co.uk from a particular seller, adds it to their basket, checks out and pays, all without visiting the seller’s own website. The seller is notified of this and dispatches the coffee table to the customer. We take a commission on the sale.

A legal minefield
Technically, this is relatively simple. But it’s a legal minefield if done the wrong way. Any business that takes payment for goods becomes accountable under the Sale of Goods Act 1979, Distance Selling Regulations, etc. Several websites such as Not On the Highstreet appear to do precisely this with home furnishings; they take payment from the customer, pay the seller a proportion of this and keep the rest. They then attempt to contract out of their statutory obligations in their terms and conditions, insisting that it is in fact the seller who is liable in the event of problems. However, in my view and that of my solicitor, if this were ever legally challenged, it would fail. Legislation to protect consumers cannot be ousted by private agreement. It’s the equivalent of those shops who put a “No Refunds” sign in the window. The situation is that, like it or not, if a company accepts payment for goods, there’s a whole load of legal baggage that comes with that. In the eye of the law, they’re a retailer and there is nothing they can do to change this.

Paying the seller, not us
The only way to remove legal liability is for the customer not to pay us at all. They must pay the seller, who then quite rightly takes on the legal burden, and we charge the seller our commission afterwards. For this to work transparently, we need to perform some technical magic such that (1) the customer’s user experience of paying feels normal and natural, even though they’re paying the seller rather than us and (2) the whole process is automated, so we don’t need to get involved in manual invoicing and placing orders for home furnishings with sellers. In this way, it feels almost identical to if the customer paid us directly, except we avoid any legal liability. We do this as follows.

Paypal API
Every seller that comes on board must set up a dedicated Paypal account. If, say, a customer buys some modern furniture on furnish.co.uk and checks out, they’re taken to the seller’s Paypal page, not ours. The customer feels comfortable with this, as they just see a Paypal page. Nothing abnormal there. As far as the customer is concerned, their work is now done. But there’s more to the technical process.

The seller’s Paypal account is configured such that furnish.co.uk gets an automatic notification through the Paypal API when a purchase takes place. This information is then automatically processed, the commission due calculated and a tally kept of commission due for each seller. Because we can also automatically monitor payments made to our own Paypal account and by who, we also know when a seller has paid us our commission. So we can calculate the total a seller owes us by taking the commission owed from all sales and subtracting from it the total paid to us. The seller is automatically emailed regularly with the outstanding balance. The email to the seller contains a link that directly opens Paypal on a page ready to transfer the sum owed into the Paypal account of Furnish.co.uk. The seller simply needs to click a confirmation to make the transfer; it couldn’t be easier for them.

In the event the seller doesn’t pay within the allocated time, we have some powerful options available. We could remove the seller from Furnish, thus cutting off their income from any furniture sales on furnish.co.uk, or even contact the customer, so it’s simply not worth it for a seller to act dishonestly. An added beauty is that a seller is welcome to top up their payments to us at any time to avoid any risk of underpayment, just like topping up your mobile phone credit.

Ensuring automatic notifications happen
A flaw in the plan is that a seller *could* secretly switch off the automatic notifications from their Paypal account such that furnish.co.uk thinks that no sales are happening when they actually are. There’s an answer here too, which we’ll be implementing shortly. Every few days or weeks, we’ll automatically make a random payment of just a few pence to each seller’s Paypal account. If their notifications are set up correctly, we’ll be notified of the payment we made and we’ll just reduce the amount owed by that seller by the few pence we transferred. But in the event we don’t get notified of the payment we made, we’ll know that the seller has switched off the automatic notifications, and we can then deal with this accordingly.

I should mention that all of the above functionality is presently in development, but it’s looking good. I’ll let you know how it goes in a future post.

AffJet = Better Affiliate Analysis

AffJet’s slogan summarises what AffJet is in three short words which is targeted at it’s market, but for those of you who don’t know what affiliate analysis refers to, here’s a quick introduction.

Affiliate marketing is a great way for websites to make money from their content and essentially, the process goes like this:

Site A displays links to Site B within their content, using a tracked or “affiliate link”. Every time a user on Site A clicks that affiliate link and performs a desired action on Site B (buys something, signs up for a service etc…), Site B gives Site A some commission as a little thank you!

This process works at it’s best when Site A and Site B are highly related to each other and, because the sales are tracked, Site B only pays for the desired actions when they happen. This differs with traditional display advertising, used online and offline, because as an advertiser you pay for all your advertising regardless of whether you receive any desired actions from it. For this reason it has taken off really well and is used by many websites across the world wide web.

Affiliate marketing uses intermediary software providers called affiliate networks who provide a platform which connects Advertisers and Publishers (aka Merchants and Affiliates). Affiliate networks allow relationships between these two parties to be built, agreements to be put in place and actions to be tracked and they also supply reporting for their merchants and affiliates.

Where does AffJet come in?

The problem with the current affiliate marketing system is that there are many affiliate networks each with lots of merchants and affiliates on their systems. As an affiliate, logging into each of these affiliate networks to find out how your websites are performing can be really time consuming, and creating reports to combine all of the data together really does take a long time.

This makes effective analysis of your data a much more daunting task than it needs to be.

AffJet provides a solution which aggregates all the data held by affiliate networks into one place. This allows affiliates to log into one place to find out how their websites are performing and saves loads of time.

What can affiliate marketers do on AffJet?

As an affiliate, you can take advantage of many useful, comprehensive and varied features within AffJet including:

• Custom Projects to allow grouping of links from any website, network or merchant
• Custom dashboards to allow you to see the metrics you want to see upon logging in
• Automatic import every night, we take the load on our servers to save your computer coping with an import
• Highly secured protection for your data, and regular back ups meaning you won’t lose out if your computer gives up
• Access your data from anywhere with an internet connection
• Allow yourself to act on the crucial difference between viewing your data and analysing it

The interface, design and features make AffJet simple to use and allow you to retrieve affiliate data and statistics quickly and simply, giving you back more time to spend on analysing and acting on the trends you have observed.

What direction is AffJet moving in now?

We are working hard on adding new affiliate networks (for greater coverage) and also new features into AffJet at the moment. A lot of these changes are being brought about by real user feedback as we want to make sure that we work on things that will make a real difference to affiliates daily work load.

We currently have a launch competition running, where affiliates using AffJet have a great chance to win an iPad regardless of their size. This competition will start when we have 50 users on AffJet and we are hoping to get a lot of feedback from these users that can help us make AffJet even better. You can read more about the AffJet iPad competition here.

Did I mention Affjet is free to use while we are in our beta testing stage?!

You can keep updated with what is going on with AffJet by following us on our AffJet Twitter account

But, in the background, there’s some seriously clever stuff going on. We’re automatically scraping stores’ websites (with their permission of course) or scanning their feeds at regular intervals, and then subjecting this data to heavy processing to determine category, style, colour, materials, etc and also doing some very leading edge stuff to make our custom-developed search engine produce super-accurate results.

For me, the most important thing about furnish.co.uk is that when a user types “wooden coffee tables” into the search or navigates to coffee tables or table lamps, that’s precisely what they get. They don’t see a coffee jug under coffee tables. I know of no sites that do this particularly well.

So, I thought I’d write about how we achieve some of this stuff – to start with, how we automatically and accurately categorise the thousands of products into our category hierarchy; how we know a bed is a bed, a wall light is a wall light, etc. I’m not the main technical guy at furnish.co.uk – that’s Alan – so I’ll keep things at a relatively non-technical level. But, hopefully, you’ll find it pretty interesting.

Categories and clones
We’ve constructed a hierarchy of hundreds of categories, covering every interiors product for the home. We did this by analysing other sites that sell home interiors products, e.g. John Lewis, Graham and Green, Marks and Spencer, Heals and then put together something that we thought worked.

We decided to let users have multiple ways to navigate through the category hierarchy to the same product for ease of use. However, for simplicity, each product only actually has a single category. That’s because we have ‘clone categories’, where a category can exist in multiple places. For example, Rooms -> Bedrooms -> Bedroom furniture -> Bedside tables shows the same items as Products -> Furniture -> Bedroom furniture -> Bedside tables. One is a clone of  the other and means the item itself only needs a single category.

Assigning categories
Based on the above, we automatically assign categories to items. For each item imported, we do the following:

1. We pull several fields out of the item that we think will give us a hint to its category and we prioritise these fields. For example, we may think that the item name is most likely to give us a hint, followed by the description. The choice of fields and their priority varies by supplier; some suppliers actually have a categories field that we can use.
2. Next, we take each of these fields in turn (highest priority first) and attempt to determine the category. We do this using a HUGE synonyms library that we’ve painstakingly put together from scratch, where each category has a set of associated synonyms. You can see a screen shot from our back end system below. Furthermore, each synonym is prioritised. So, the system finds all synonyms contained within the field that it’s analysing, but then chooses the one with the highest priority. There are also negative synonyms, i.e. where an item cannot be a certain category if it contains certain words. In the event the field contains multiple synonyms with equal priority, it uses the one that comes first in the field.
3. If there are no matches, it moves onto the next field within the item to see if that contains a synonym.
4. This is repeated for all items being imported. On rare occasions, no synonyms are found and items end up with no category. However, these are not published.

We’ve found that our processing and synonyms library is sufficiently good that any new supplier coming on board tends to get 95% correct categorisation. We then manually tweak the synonyms to ensure 100% accuracy for the supplier.

All pretty elaborate, but the result is clear and accurate navigation. We also use some of this information for the search index, but more on that in a future post.

Hello, my name is Alex Buell, I am profoundly deaf, and work as a Linux
system administrator within the Fubra infosphere. I usually spend most
of my time working on open source projects, giving back to the
community extra value in tools that allow us to do our job.

What are sitemaps?

They provide a way for webmasters (people who run
websites) to give out information about the content on their websites.
Search engines (i.e. www.google.co.uk) look (‘crawling’) through
websites to build up indexes to allow people to search for things that
they are interested in looking for.

Essentially a sitemap is just a file containing URLs in XML format,
along with additional metadata about each of these URLs (when was it
last updated, how often does it change, how vital it is), enabling
search engines to make intelligent decisions about searching for links
to or from pages on the web sites.

Sitemaps have to be regularly updated so we have a tool that
automatically generate sitemaps which are stored within the web site,
search engines can download these files and read through all the URLs.

A new tool for validating sitemaps

We at Fubra have developed a web tool to help with validating sitemaps
on our websites. The tool does two things; reads the sitemap files off
the website, and presents to us in a human readable form, and if asked,
check through the URLs, displaying the http code (i.e. 404, 301, or 200)
in the status icons on the right side.

How to use the Validator webtool

To use the tool, point your browser at the
Sitemap Validator site, and type in the URL
(i.e. www.talkfootball.co.uk), and see what the tool does with it.

As it stands, the validator tool will look through the URLs and
its associated metadata for validity and colour the icon on the right
accordingly. Red icons means that the URL is a duplicate, orange means
the date and time associated with the URL is invalid, whilst green
means that the URL and its metadata is valid. The tool sorts the URLs
in a way that all invalid URLs comes first and all valid URLs comes
last. This makes it easy to see what is wrong with the sitemap.

Additionally, there is a button ‘Check’ in the top row, which, if
clicked upon, will run a check on all the URLs on the site, and display
the http code.

Any other business?

All comments, flames and feedback welcome, please do drop me an email
at alex at fubra dot com

We have been quiet for a while on the subject of the mac minis we installed into LINX at Telehouse several months ago…

You may remember the previous article, basically we are using a pair of Mac Mini computers to connect our hosting platform to the LINX Internet exchange in London.

Having a connection to LINX enables us to get much closer to our end users and to our primary content providers, meaning our information moves faster and at lower cost.

Our decision to use the Mac Minis as routers was based on a combination of economic factors. Compared with the typical Cisco 7200 or Juniper M-series routers, the Minis use less space and a fraction of the power whilst providing a massive 2GB of RAM for routing tables, a Core2 DUO 1.83GHz CPU and Gigabit network interface for a tiny fraction of the cost.

Whilst some old-school network purists might have their own opinion on software routers, we have used Linux based routers at our edge and core for over four years now with no issues at all and saved tens of thousands of pounds in routing hardware. So far, packet-for-packet the Minis have performed every bit as well as our 1u rackmount routers in use at our less power constrained sites.

Now that the initial setup is completed, I have been asked to into go more detail about the system and how we solved the various issues encountered along the way. And there have been many issues to solve, mainly because the Mac Mini wasn’t really designed for use as a border router.

So first, the hardware issues…

Hardware problem 1 – Booting without a monitor

The Mac Mini is designed for use as a desktop computer. As such it expects to have a monitor attached – and somewhat unusually will not boot without one.

To get around this problem we made up some svga dongles as described here and connected them using the svga->dvi adapters supplied by Apple.

Granted, from a purist perspective an svga dongle sticking out of the back does spoil the form of the Mini slightly, but in another way it looks like a chromed exhaust pipe, quite fitting for our little hot-hatch routers!

Hardware problem 2 – Automatic power-on

The next hardware problem encountered was how to make the minis boot automatically after a power cut/cycle. Fortunately this was also figured out quite easily, it was simply a case of adding the following line early in the boot process:

  # setpci -s 0:1f.0 0xa4.b=0

Hardware problem 3 – complex routing on a single network interface

The final hardware problem – well not really a problem, but something I often get asked about – is how the networking works. The Mini has just one network card and most people think of a router as having more than one physical network interface.

However, by installing a decent ‘managed’ network switch between our physical network connections and the router, we are able to use VLAN technology (802.1q VLAN Tagging to be precise) to provide multiple secure ‘virtual’ network connections to Linux.

Problem solved. Next, software and driver issues.

We run Ubuntu-Server 7.04 as the operating system on the Minis. For those who aren’t in the know yet, Ubuntu is one of the leading Linux software distributions. Fubra is no stranger to Linux with more than 150 Fedora, Centos and Ubuntu Linux systems deployed across the network as workstations, web servers, email servers, databases and routers.

Software problem 1 – Dodgy ethernet drivers…

Probably the biggest problem we had with the Minis was getting the network cards to remain stable under heavy load. The culprit in this case was what appears to be the buggy (aka “experimental” in kernel speak SKY2 network card driver supplied in the Linux kernel.

In the end we decided to stop using the SKY2 driver altogether until it is more stable. Instead we chose to use the Marvell supplied driver which can be found here. Since switching to the Marvell driver the network card has remained rock solid and we have had no problems or device lockups at all.

Software problem 2 – Connecting to the Internet. Oh yeah, and support for md5 passwords…

For most people, connecting to the Internet means subscribing to a broadband service, which typically provides a single route out to the Internet at between 2 and 20Mb/s. As a hosting company with a lot of high-traffic websites, we have multiple connections to the Internet running at between 100 and 1000Mb/s, enabling us to handle a lot of simultaneous visitors without slowing down.

The other main reason for having multiple connections is that we can survive ISP failures on our network, but implementing this level of protection requires another software protocol running on the router called Border Gateway Protocol, or BGP.

BGP is like a route-planning system for the Internet. In much the same way as an in-car, GPS-based navigation system does, it knows of every route between any two given points on the Internet and can calculate which is the best path to take.

The only issue we had with BGP was getting md5 signatures (a common authentication method for BGP routers) to work – and judging by the Quagga mailing list and wiki this has been a perennial problem for quite some time. I won’t go into too much detail but from what I can tell the resolution was quite typical for an open source project:

1. Some kernel developers suggest md5 signing has no place in the kernel – it should go into userspace.
2. Some userspace developers feel the most appropriate place for md5 signing is in the kernel.
3. Both solutions end up developed… [bgpmd5.pl] [md5qd] [Kernel >=2.6.20]

In the end we went for the following recipe:

• Linux Kernel 2.6.22 (compiled from Vanilla sources)
• Quagga 0.99.9 plus Solinno patch found here

Note it took a couple of extra steps not all that well documented anywhere to get quagga to work properly with the in-kernel support for md5 signing:

First, since the patch was taken against quagga-0.99.8, it has to be applied as follows:

  # tar -zxvf quagga-0.99.9.tar.gz
  # cd quagga-0.99.9
  # patch -p1 <../quagga_md5_bsd_linux_v7.diff

Next, the sources need to be configured with an option to disable IPV6 (note that this disables support for IPV6 in Quagga altogether):

  # ./configure --enable-ipv6=no --localstatedir=/var/run/quagga --sysconfdir=/etc/quagga

To enable the md5 support provided by the Solinno patch, an extra line should be added to config.h before compiling and installing:

  # echo "#define HAVE_TCP_MD5SIG" >>config.h
  # make && make install

I’d like to thank all the guys on the Quagga mailing list who have worked on the md5 problem over the last couple of years.

Having used both of the user-space solutions in production (bgpmd5.pl, md5qd), I can honestly say that the in-kernel method seems to be far more reliable once you get it working and “just feels right”.

Plus a special shout to Dunc from The Bunker who has been working on the same bgp md5 problems for just about as long as I have

So how fast will our little hot-hatches go?

To be fair, I’m not yet 100% sure what we can expect from our Mini routers but the bottom line is, with a 1.8GHz processor, 2GB of ram for holding large routing tables, a Gigabit network interface and a liberal sprinkling of kernel tweaks I fully expect the routers to handle several hundred Megabits per second (and yes, before you shout at me I know – its really about packets per second, not Megabits!).

Hopefully we’ll get the opportunity to fully test some Minis under lab conditions to get some real numbers. When we do, I’ll be sure to post the results in a subsequent article.

I’m convinced that one day we will have to invest in something a little more substantial, after all a hot hatch is great but it doesn’t have enough room for four kids and a pram.

Until then we’ll enjoy the lower fuel consumption and lower purchase cost of our Mac Minis.

When Paul showed me WinSCP as a great alternative to FTP I was well chuffed mainly because we haven’t really been able to get to our servers via FTP as hardly any run FTP servers but also because I can use my SSH key to authenticate rather than remembering passwords which I almost never use because they are so hard to remember.

The people behind this project should be proud it is definitely one of my favorite apps.