Last month I put up a new gardening website. It’s a fully dynamic site without one single “?” in any of the URLs, thanks to Mod Rewrite. The site started out with about 250 pages and now is up to 500 pages. I pointed a number of links to the new site from other sites with high PR to get it spidered quickly. Here’s what happened.

MSN’s spider was the first to visit, and quickly indexed about 60 pages. MSN’s spider has continued to return and now, after 4 weeks, has spidered and indexed almost the entire site. MSN was also the first search engine to display pages in the SERP (search engine ranking pages) for a good number of search phrases. A good number of these search phrases are showing up in the top ten results, so a very nice return on well SEO’d pages.

Google’s spider was the second to visit and has been incredibly slow at spidering pages. We even set up a Google Sitemaps account and submitted a full xml sitemap, and this still didn’t help much. After 4 weeks Google has now spidered well over 300 pages, but only 62 pages show up in their index. To date very few keywords show up in the SERPs, and I’m looking through ten pages of results.

Yahoo’s spidering has, by far, been the slowest to spider and index pages. To date there are only 90 pages indexed in Yahoo, and almost all of the pages do not show up in SERPS.

For webmasters looking to get their pages into the search engines, and show up in the actual SERPs, MSN is the clear winner. Google comes in at a far second and Yahoo a wait-forever third.

The funny thing about all this is that I have Google adsense on almost all of the pages on the gardening site. If Google is taking forever to spider and index pages with Google asense on them they are potentially losing millions every day in lost click revenue. Whatever. I think Google has it’s finger in so many pies now the core product is falling behind.

• SEO Tools - Web CEO Review by Adam Sorensen
• Google spidering and indexing pages in hours
• SEO Friendly Flash Design

This entry was posted on Monday, May 15th, 2006 at 12:22 pm and is filed under Content Optimization, SEO. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.