Yahoo upgrading crawler Slurp
For those who don’t know this already, search engine crawlers are software used to “spider” websites throughout the Internet. Monitoring crawlers’ behavior is part of the SEO firm’s routine.
According to Yahoo search team leaders David Simpson and Priyank Garg, their new robot is going to be more efficient in crawling websites which will result in a decreased number of hits and bandwidth consumption of approximately 25%. The shift was performed last week.
This event would certainly be welcomed by most webmasters and marketers since the Sunnyvale, CA based company has been hitting their web pages pretty heavily, sometimes without any apparent logic.
A short note for the beginners: when preparing a web analytics report you should always discount visits from the automated tools mentioned above – these are not real visitors and some analytics solutions’ counts could be confusing. For example, leading log analysis provider Webtrends requires custom configuration to keep out robot visits from a report.
Another side effect coming along with the Yahoo Slurp’s upgrade is a slight ranking fluctuation. No major change should be noticed unless it comes together with an algorithm update. Most likely affected will be the “freshness” of results. Yahoo has been known for providing sometimes quite outdated information to searchers.
The Sunnyvale Company seems to also be working on some ranking factors, very similarly to MSN’s recent algorithm changes. The trend is to mimic Google’s way by emphasizing the weight carried by certain websites. Not all sites are given the same credibility, based on 100’s of factors on and off their domain. For example a .gov site is a trusted source. As a matter of fact, .edu’s are as well although this is currently arguable since SEO’s are using Universities’ sites often for artificially increasing rankings of other websites. Some colleges have even been selling text links for quite some time but this will be described along with its outcomes in a later blog post.
More details about Yahoo search crawler, named Slurp can be found there for those who would like to understand more. Understanding how search engines crawl your site is only the beginning of understanding how they work – this may come as a good advice for some self proclaimed SEO’s who still think that the whole power of SEO is in so-called SEO copywriting.
In the meantime keep watching all 3 major search engines, they are preparing major changes towards the end of August as it seems. Well, so it uses to be each year by September.
