Search Engine Spiders And Your Robots.txt File
47706
Internet > Web Development
Oct 11, 2007
eatmoreherbs
Search Engine Spiders And Your Robots.txt File
In this article we will discuss search engine spiders and what they do. You will also learn how to create a robots.txt file and why you might need one. Search engine spiders are automated software programs that crawl the Web looking for pages to feed to search engines. They are also called crawlers, robots and bots. Spiders are one of the most useful programs on the internet. They are a key part in how the search engines operate. Spiders allow your site to be found by the millions of people who use search engines. Feed the spiders right and they will tell the search engines about your site. How Spiders Work A search engine is an index to the Internet, search engines point to relevant web sites depending on your search. Search engines need a tool that is able to visit websites, navigate the websites, decide what the website is about and add that data to the search engine. Spiders are essentially programs that "crawl" sites and report back to their boss their findings. Their purpose in life is to make it easy for your site to get listed in search engines. Spiders work by finding links to web sites, visiting those web sites, going through the content of a web site and then reporting the content of the site back to the database of the search engine they work for. From there, the information is added to the search engine, and the site then shows up in search results. The robots.txt file By defining a few rules, you can tell robots to not crawl certain directories or files, within your site. Web sites do not absolutely have to have a robots.txt file, they can get along just fine without one. Most spiders look for a robots.txt file as soon as they arrive on your site. Take a look at your site statistics. If your statistics has a "files not found" section, you may see many entries where spiders failed to find the file on your site. The default behavior is to allow all unless you have a Disallow for that resource. If you wish to exclude some of your pages from search engine indexing, this is the tool approved by the search engines. Creating a robots.txt file that guides spiders is simple. If you want to allow the spiders to crawl your site but exclude directories of your choice, copy and paste the following into a blank txt file: User-agent: * Disallow: /directory1/ Disallow: /directory2/ Disallow: /directory3/ To exclude files of your choice, type in the path to the files you want to exclude: User-agent: * Disallow: /directory1/page1.html Disallow: /directory2/page2.html Disallow: /directory3/page3.html To exclude all the search engine spiders from your entire web site, copy and paste the following into the txt file: User-agent: * Disallow: / This will keep a specific search engine spider from indexing your site: User-agent: Name_of_Robot Disallow: / To allow a single robot and exclude all other robots: User-agent: Googlebot Disallow: User-agent: * Disallow: / There can only be one robots.txt on a site, and you may not have blank lines in a record. Once you have it the way you want, save the file as "robots" and as a .txt file. Uploading the file to the root directory of your site, that is the directory where your home page or index page is. Put the robots.txt file right alongside the index file.
Sign up for the Web Success Weekly Email. Lean simple, step by step methods to get your business online and making money, the easy way: http://websuccess.info/ By Harvey Lew Robinson: http://websuccess.info/seo/spiders.html
send email to eatmoreherbs
Write a Review
Add to My Favorite
Refer it to Friend
Report Article
Other links owned by this user
In this article we will discuss search engine spiders and what they do. You will also learn how to create a robots.txt file and why you might need one.
Category:
This article is a little advise about how do you can develop an well designed and profitable email marketing campaign.
Category:
In this article we'll go over what a blog is, a short history of blogs, what they are used for and why you may want one.
Category:
To get listed in Google you need for the Googlebot to visit your site. This Article will show you how to do that.
Category:
It sometimes happens that a search engine spider don't visit your. This article is about how to get the spiders to visit your site.
Category:
This article goes over the basics of the title and description meta tags and how to write them.
Category:
Most herbs are tough wild plants which thrive when pampered by gardeners.
Category:
After you've read this, you'll know what techniques will get you in trouble and which ones are safe to use.
Category:
Getting listed quickly in the major search engines is what every web site owner wants to do. This article will help you do just that.
Category:
Places you can submit your links to compete for keyword position in the major search engines.
Category:
Other links at Internet > Web Development
Seo for those with small budgets will take some time and should be a long term effort, but the benefits are many. If you don't have the cash to manage using a service or one of the alleged experts, don't worry. You can do this!
Category:
Video marketing, as we know it is starting to revolutionize the way we view content on the Internet. With video sharing sites such as Yoube, Revver, and Break springing up clone sites daily, it is no wonder that people are looking to view video more than
Category:
You'll always have bigger and better competitors, but that doesn't mean you can't succeed. However, to deliver your message to the maximum number of people out there, to effectively compete with the competition, you first need to get your website ranking
Category:
Internet marketing is a challenge, right? Well, it can be if you don't use the right internet marketing tools. This article will help you find the tools you need to increase your site's targeted traffic.
Category:
Pay per click advertising can be costly for people who do not know how to use it. Here's how you can use pay per click advertising affordably, while getting maximum exposure for your site. Here's how.
Category:
Site Sponsor
Directory Statistics
Articles: 68285
Categories: 501
Yahoo Entertainment
