Typically we have a tendency to rank well on one engine for a specific keyphrase and assume that all search engines can like our pages, and hence we tend to will rank well for that keyphrase on a range of engines. Unfortunately this can be rarely the case. All the majorsearch engines differ somewhat, thus what is get you ranked high on one engine could actually help to lower your ranking on another engine.

It is because of this that some folks prefer to optimize pages for each particular search engine. Usually these pages would only be slightly totally different but this slight distinction might make all the difference when it involves ranking high.

However as a result of search engine spiders crawl through sites indexing each page it will find, it might come across your search engine specific optimizes pages and as a result of they are terribly similar, the spider might suppose you’re spamming it and will do one amongst 2 things, ban your site altogether or severely punish you in the form of lower rankings.

The answer is that this case is to stop specific Search Engine spiders from indexing some of your net pages. This is often done employing a robots.txt file that resides on your webspace.

A Robots.txt file is a important part of any webmasters battle against obtaining banned or punished by the search engines if he or she styles totally different pages for various search engine’s.

The robots.txt file is just a simple text file because the file extension suggests. It’s created employing a straightforward text editor like notepad or WordPad, sophisticated word processors such as Microsoft Word can only corrupt the file.

You can insert certain code in this text file to form it work. This is often how it will be done.

User-Agent: (Spider Name)
Disallow: (File Name)

The User-Agent is that the name of the search engines spider and Disallow is that the name of the file that you do not want that spider to index.

You have got to start a replacement batch of code for each engine, but if you want to list multiply disallow files you’ll be able to one beneath another. For example

User-Agent: Slurp (Inktomi’s spider)

Disallow: xyz-gg.html
Disallow: xyz-al.html
Disallow: xxyyzz-gg.html
Disallow: xxyyzz-al.html

The higher than code disallows Inktomi to spider two pages optimized for Google (gg) and 2 pages optimized for AltaVista (al). If Inktomi were allowed to spider these pages in addition because the pages specifically created for Inktomi, you will run the chance of being banned or penalized. Hence, it’s always a good idea to use arobots.txt file.

The robots.txt file resides on your webspace, but where on your webspace? The foundation directory! If you upload your file to sub-directories it can not work. If you needed to disallow all engines from indexing a file, you just use the “*” character where the engines name would typically be. However beware that the “*” character won’t work on the Disallow line.

Here are the names of some of the massive engines:

Excite – ArchitextSpider
AltaVista – Scooter
Lycos – Lycos_Spider_(T-Rex)
Google – Googlebot
Alltheweb – FAST-WebCrawler

Be positive to test over the file before uploading it, as you may have made a simple mistake, that may mean your pages are indexed by engines you do not wish to index them, or perhaps worse none of your pages would possibly be indexed.

Are you looking for more information on website seo. Or about website seo. Get pro advice on website seo.

Tags: search engine, Search Engine Optimization, search engine services, SEO, website seo

Related Posts

• How To Create A Robots.txt File
• The Quick Way To Increase Your Google Ranking With An Effective Google Sitemap
• A Link Checker Is Now a Critical Component In Your SEO Arsenal

This entry was posted on Wednesday, March 3rd, 2010 at 5:03 pm and is filed under SEO. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.