.: How to keep robots out of your web site
By:Dr. Bonomi
Category:Home / Internet Marketing / Search Engine Optimization
THE ROBOTS.TXT FILE
You know that search engines have been created to help people find information quickly on the Internet, and the search engines acquire much of their information through robots (also known as spiders or crawlers), that look for web pages for them.
The spiders or crawlers robots explore the web looking for and recording all kinds of information. They usually start with URL submitted by users, or from links they find on the web sites, the sitemap files or the top level of a site.
Once the robot accesses the home page then recursively accesses all pages linked from that page. But the robot can also check out all the pages that can find on a particular server.
After the robot finds a web page it works indexing the title, the keywords, the text, etc. But sometimes you might want to prevent search engines from indexing some of your web pages like news postings, and specially marked web pages (in example: affiliate´s pages), but whether individual robots comply to these conventions is pure voluntary.
ROBOTS EXCLUSION PROTOCOL
So if you want robots to keep out from some of your web pages, you can ask robots to ignore the web pages that you don´t want indexed, and to do that you can place a robots.txt file on the local root server of your web site.
In example if you have a directory called e-books and you want to ask robots to keep out of it, your robots.txt file should read:
User-agent: * Disallow: e-books/
When you don´t have enough control over your server to set up a robots.txt file, you can try adding a META tag to the head section of any HTML document.
In example, a tag like the following tells robots not to index and not to follow links on a particular page:
meta name="ROBOTS" content="NOINDEX, NOFOLLOW"
Support for the META tag among robots is not so frequent as the Robots Exclusion Protocol, but most of major web indexes currently support it.
NEWS POSTINGS
If you want to keep the search engines out of your news postings, you can create an an "X-no-archive" line in of your postings' headers:
X-no-archive: yes
But although common news clients, allow you to add an X-no-archive line to the headers of your news postings, some of them don´t permit you to do so.
The problem is that most search engines assume that all information they find is public unless marked otherwise.
So be careful because though the robot and archive exclusion standards may help keep your material out of major search engines there are some others that respect no such rules.
If you're highly concerned about the privacy of your e-mail and Usenet postings, you must use some anonymous remailers and PGP. You can read about it here:
http://www.well.com/user/abacard/remail.html http://www.io.com/~combs/htmls/crypto.html
http://world.std.com/~franl/pgp/
Even if you are not particularly concerned about privacy, remember that anything you write will be indexed and archived somewhere for eternity, so use the robots.txt file as much as you need it.
Written by Dr. Roberto A. Bonomi
Article keywords: robots, robots.txt, robots exclusion protocol, marketing internet marketing, home business
Article Source: http://www.articles32.com
Dr. Roberto Bonomi is a successful e-book writer that shares his home business experience at: www.easy-home-business.com If you already have, or are looking for an Internet Home Business, you can't miss the free knowledge that you'll receive at his site, and you can post free your own articles at articles.drbonomi.com
.: New Search Engine Optimization Articles
1). How to Improve Search Engine Ranking
Powerful 4-step process to improve search engine ranking
2). How to Make Websites that Search Engines Love
How you can apply the fundamental basics to create a website that the search engines will love to spider.
3). Articles Are Seeds of Knowledge - A Biblical Look at Duplicate Content
Article Marketing compared to the successful distributon of the Christian Bible.
The Christian Bible is the Worlds most successful publication, with over 6 billion copies printed, similar to the total World Population.
4). Content Management Systems Equal Business Suicide!
One of the fastest way to minimise your chances of web business success is to use a Content Management System (CMS).
5). Importance of Search Engine Marketing Firms
Literally speaking, there is no business in todays world of World Wide Web, which doesnt have online business. To reach ever growing population of internet all across the world it is mandatory to have a good online presence for any business. Moreover, meeting customers online is the cheapest possible way.
6). Internet Search Engines, Important Details Everybody Should Know
Internet Search Engines see something totally different than what we see when we look at a webpage. It is all in the webpage code. Have you ever seen webpage code? Place one of your webpages into your browser.
7). Why Websites Get Banned From Search Engines
These are methods and techniques that end up getting websites banned from Google and other search engines.
.: Top Search Engine Optimization Articles
1). SEO Contests: Good or Bad?
As a webmaster you probably already know what a SEO Contest is or you surely came across some of them or even participated in. Some of the major SEO contests organized are the v7ndotcom elursrebmem (by v7n.com), Ambachdotcom, Carcasherdotcom, and more recently the Cpayscom2 Online Casino where the winner gets $10,000 for ranking number one on MSN. Now the big question is whether it is good or bad to organize SEO Contests.
2). High Paying Lateral Keywords
If you run AdSense on your site you know that some words pay more than others, much more in fact. More than likely you have also learned that terms like "structured settlements" and "mesothelioma" can produce incredibly high PPC revenue, if they show up on your site at all. Unfortunately hundreds of thousands of other webmasters are "on" to this practice judging by the number of sites created regularly to capitalize on the phenomenom.
3). Is Your Website Invisible? The Google Sandbox Solution
What is the Google Sandbox? How Do You Know if You’re Stuck in the Sandbox? Does the Sandbox Really Exist, or is it Just the Google Algorithm? Why Do You Get Sandboxed? Is There a Way to Trick the Sandbox Filter?
4). An SEO Glossary - Common SEO Terms Defined
Search Engine Optimization (SEO) has become an essential weapon in the arsenal of every online business. Unfortunately, for most business owners and marketing managers (and even many webmasters), it's also somewhat of an enigma. This is partly due to the fact that it's such a new and rapidly changing field, and partly due to the fact that SEO practitioners tend to speak in a language all of their own which, without translation, is virtually impenetrable to the layperson.
5). Creating Sitemaps For Google, MSN AND Yahoo! - The Easy Way
If you own or maintain a website or intend to own one, wouldn’t it be great if you get frequent visitors who find satisfaction in getting exactly the information they need from your page?
While that satisfaction largely depends on the contents of your website, how you get to be accessed by website users is the most critical factor of website development.
6). How to Make a Sitemap For Your Website In Five Steps.
A website's sitemap can be compared to the table of contents of a book. The sitemap is important because it guides visitors to the parts of the website in which they are interested. The sitemap allows surfers to reach their destination without wasting time.
Sitemaps are also important from the point of view of search engines. If the search engine finds a sitemap it is much more likely that your page will be fully visited (spidered) and indexed; allowing web surfers to find your pages in their searches.
7). 301 Redirect - The SEO way to rename or move files or folders
In this article I will discuss page redirection techniques, what works and what to avoid.
What is page redirection and why would you want to use it?
Let’s say you rename a page on your website, for whatever reason. Perhaps you decided to revamp your entire naming convention, perhaps you decided to restructure your site and need to move pages into different folders, or you just realized that you are missing valuable keywords.