So much for a Search Engine race

I’ve just finished watching a re-run of the BBC’s Top Gear. Richard Hammond took on an RAF Eurofighter in a Bugatti Veyron in one of their classically contrived races.

The Eurofighter came in first, but the Veyron wasn’t too far behind. It was a race of sorts, give or take. I only wish I could say the same for my attempt at search engine spider racing.

Google came in first by a country mile, with a complete indexing done in about 84 hours. We’re 10 days, a full 240 hours, into the race now and Yahoo has managed to get a grand sum total of one page indexed.

As for Bing. Well.

Bing is hanging out down at the start line with it’s eye candy interface clinging onto some pages that have not existed on this domain for at least two years.

While it is possible that Microsoft have developed a time machine, I think it’s more likely that msnbot doesn’t know an http 404 response from a mouse pad. Combine that with an inability to honour robots.txt and I’m not sure the folks up in Seattle know for sure if they’re running a search engine or a cake stall.

There has been a buzz in the blogsphere about real time search for a while, with twitter leading the charge in delivering on the dream. Twitter of course has the advantage that all the content it needs is provided on it’s doorstep by hordes of twittering users.

Back in the world of conventional search engines the battle to gather content is fought by the spiders. Clever robots sneaking around the web on the constant lookout for new or changed stuff. Indexing, ranking, summarising. The unsung heroes in our digital world even.

No prises for guessing how poor the real-time search ability of Bing is going to be if it takes longer than 10 days to index data that was handed to it on a platter, and 2 years to remove content that has been returning a 404 for that long.

My website is an internet backwater, I’m quite realistic about that little detail, but if Google pays attention to me, I’ll focus my SEO attempts on Google and ignore the other bit part players for the time being.

Leave a Reply