How Search Engine Spiders Work
6991
Internet > SE Tactics
Mar 1, 2007
Anton Cheranev
How Search Engine Spiders Work
There are hundreds of search engines available today, but some are far more complex than others. This article will give you an overview of how some of the most popular ones work.
Let?s start with a smaller engine: InfoSeek. They only index about 200 words of your web page, so it?s important to make sure that you have meta tags on your site, and that the most important things are listed first. The information you put in your meta tags will be used to display a description of your site, and most meta tags can contain about 200 characters of text. The keywords meta tag, however, can have up to 1,000 characters.
These simple rules are important to keep in mind for all search engines. The more important that the information is, the closer it has to be to the beginning of your meta tags or even the beginning of your site?s content. Many search engines won?t even touch your meta tags so it is important that you have the same information in your body that you have in your meta tags (although you obviously cannot simply enter lists and lists of key words as this would be detrimental to your site?s content).
The AltaVista search engine will send Scooter, its spider, to check out your entire site. Scooter can take as long as three months to spider and fully index your site ? the average spider only takes 6-8 weeks. Scooter will normally spider somewhere between two and ten pages from your site each week. This means that the longer that your web site lasts, the better it will be indexed which is in example of how search engines implement Darwin?s Theory into their ideology.
Excite used to be a search powerhouse, but has now been dropped as the provider of AOL and Netscape search, so it?s less important than it once was. The algorithm it uses to determine keyword relevance is very complicated: it indexes your pages and then attempts to summarize them by selecting only the most relevant sentences. Expect to have your pages reviewed roughly once every two weeks. Keep in mind, though, that with meta tags have no meaning to Excite when it comes to rankings, even though it will use your description tags as long as the words are relevant to your pages? content.
Let?s move on to Lycos. Lycos has fully integrated the Open Directory Project (ODP) into their mainstream results pages, and they also use search results from AllTheWeb. Lycos also runs click-throughs to their sister site HotBot. Lycos is one of the harder search engines to understand, as their submission pages say one thing but then they index your site in a completely different way. As a general rule of thumb, your site will be indexed in Lycos in due time as long as you get indexed in ODP and AllTheWeb.
Even though WebCrawler is owned by Excite, it still has its own search engine and indexer. If you happen to be listed with WebCrawler, you should try to stay listed with them, as it isn?t the easiest search engine to get listed with. Its hit-and-miss standards combined with the sporadic indexing methods makes the submission process tough, although not impossible.
The biggest player is, of course, Google, who use a page ranking system as the central basis of their index. It was once nearly impossible to manipulate this page ranking system to drive up your rankings, but people quickly figured out that the more links they could generate to their site on the rest of the net, the better Google ranked them. Google is not thought to be using context-sensitive rankings. Context-sensitive information is used at Yahoo, Looksmart and the ODP, however, and Google regularly spiders those sites when it re-indexes its own database.
MSN is another important search engine. The holy trinity of search engines at the moment is Google, Yahoo!, and MSN. These three search engines combine to provide you with the vast majority of the traffic that you will receive from search engines. MSN will generally be the first search engine to index your site and it will almost certainly list the most pages the fastest.
Although no-one can tell you exactly when you will be indexed on any search engine, it?s best to check back at least weekly. Whatever you do, though, don?t re-submit your site more often than every two months or so ? you might not get indexed at all if you do this.
Let?s start with a smaller engine: InfoSeek. They only index about 200 words of your web page, so it?s important to make sure that you have meta tags on your site, and that the most important things are listed first. The information you put in your meta tags will be used to display a description of your site, and most meta tags can contain about 200 characters of text. The keywords meta tag, however, can have up to 1,000 characters.
These simple rules are important to keep in mind for all search engines. The more important that the information is, the closer it has to be to the beginning of your meta tags or even the beginning of your site?s content. Many search engines won?t even touch your meta tags so it is important that you have the same information in your body that you have in your meta tags (although you obviously cannot simply enter lists and lists of key words as this would be detrimental to your site?s content).
The AltaVista search engine will send Scooter, its spider, to check out your entire site. Scooter can take as long as three months to spider and fully index your site ? the average spider only takes 6-8 weeks. Scooter will normally spider somewhere between two and ten pages from your site each week. This means that the longer that your web site lasts, the better it will be indexed which is in example of how search engines implement Darwin?s Theory into their ideology.
Excite used to be a search powerhouse, but has now been dropped as the provider of AOL and Netscape search, so it?s less important than it once was. The algorithm it uses to determine keyword relevance is very complicated: it indexes your pages and then attempts to summarize them by selecting only the most relevant sentences. Expect to have your pages reviewed roughly once every two weeks. Keep in mind, though, that with meta tags have no meaning to Excite when it comes to rankings, even though it will use your description tags as long as the words are relevant to your pages? content.
Let?s move on to Lycos. Lycos has fully integrated the Open Directory Project (ODP) into their mainstream results pages, and they also use search results from AllTheWeb. Lycos also runs click-throughs to their sister site HotBot. Lycos is one of the harder search engines to understand, as their submission pages say one thing but then they index your site in a completely different way. As a general rule of thumb, your site will be indexed in Lycos in due time as long as you get indexed in ODP and AllTheWeb.
Even though WebCrawler is owned by Excite, it still has its own search engine and indexer. If you happen to be listed with WebCrawler, you should try to stay listed with them, as it isn?t the easiest search engine to get listed with. Its hit-and-miss standards combined with the sporadic indexing methods makes the submission process tough, although not impossible.
The biggest player is, of course, Google, who use a page ranking system as the central basis of their index. It was once nearly impossible to manipulate this page ranking system to drive up your rankings, but people quickly figured out that the more links they could generate to their site on the rest of the net, the better Google ranked them. Google is not thought to be using context-sensitive rankings. Context-sensitive information is used at Yahoo, Looksmart and the ODP, however, and Google regularly spiders those sites when it re-indexes its own database.
MSN is another important search engine. The holy trinity of search engines at the moment is Google, Yahoo!, and MSN. These three search engines combine to provide you with the vast majority of the traffic that you will receive from search engines. MSN will generally be the first search engine to index your site and it will almost certainly list the most pages the fastest.
Although no-one can tell you exactly when you will be indexed on any search engine, it?s best to check back at least weekly. Whatever you do, though, don?t re-submit your site more often than every two months or so ? you might not get indexed at all if you do this.
Article Source: http://www.articlerich.com
Do You Want Your Own Digital Detective That Investigate Your Competitors' Websites And Reveal All Necessary Information To Dominate Over Them? If Yes, Check Out Here => www.TheSeoDetective.com
Write a Review
Add to My Favorite
Refer it to Friend
Report Article
Other links owned by this user
Offering free things to your website visitors is one marketing method that often results in a lot of sales. Free courses that are delivered via email are very popular, and people sign up for such courses on a regular basis to learn more about a topic of
Category:
As an Internet marketer, it is vitally important to know how well your advertising campaigns are doing. Advertising campaigns cost a lot of time and money, and campaigns that are not doing well need to either be changed or scraped. When using
Category:
If you?ve looked at the prices of autoresponders that are available online, you may have decided to search for and use a free autoresponder for your marketing needs. Using free autoresponders is acceptable in certain situations, and in the world of
Category:
Have you ever walked into a store in your town, and been addressed by name? This has probably happened to you at stores that you frequent often. The shop owner knows your name, and uses it. He remembers you, and he wants you to know that he cared enough
Category:
Providing a newsletter for your opt-in list subscribers provides many benefits in terms of driving traffic into your site as well as boosting the sales and profits of your site and company. This is a g ploy that will not hugely dent your
Category:
Many newcomers to the Internet marketing arena are not aware that there is a vast difference between an autoresponder program and an autoresponder service. Not knowing the difference, they often purchase the wrong type of autoresponder, and find out too
Category:
When used sensibly, website link exchanges can draw an exceptional amount of website traffic and greatly increase your search engine outcome. Nevertheless, remember that too many link exchanges can be harmful. You do n want your website clogged with
Category:
Ideally, when you perform customer service, it is done on a one-on-one basis with each of your customers. That works quite well in the offline world ? but on the Internet, that simply will not do. Your customers are literally all over the world, and
Category:
If you are using your autoresponder to sell a product or service, you must be very careful as to how you approach your potential customer. Few people like a hard sale, and marketers have known for years that in most cases, a prospect must hear your
Category:
Many people wonder how search enges really work. Although the details are complex, this article aims to give you some sight to the process without gettg too technical. Read on... Most search enges have three parts: a crawler, an
Category:
Autoresponders are one of the most important marketing tools that you can have if you are doing business online. In fact, the only thing more important that the autoresponder is your opt-in ! But all autoresponders start out without a ? the
Category:
When people place orders through your website, it is always a good idea to immediately acknowledge the purchase. One way to do this is with the use of an autoresponder that can be integrated with your shopping cart. In fact, many shopping carts that are
Category:
Building customer interest and excitement is the first step to successfull marketing man products. Autoresponders pla a vital role in building this interest and excitement. For instance, if ou were developing an ebook, ou ma want to start telling
Category:
Making sure your website has a steady amount of activity is one of the most important aspects to maintaining a successful website. Increasing website activity is one of the easiest ways you can make your site more popular as well as help your business
Category:
There are many uses for an autoresponder. One such use is the publishing and distribution of an ezine or newsletter. Since autoresponder services usually are set up to send automatic responses to the people on the list at certain intervals after they
Category:
There re hundreds of serch engines vilble tody, but some re fr more complex thn others. This rticle will give you n overview of how some of the most populr ones work. Let?s strt with smller engine: InfoSeek. They only index
Category:
Other links at Internet > SE Tactics
Search engines are one of those things in life that can't be controlled. Life is full of things that can't be controlled. We can't control who we're related to, or whether or not the sun will shine and we certainly can't control how other people on
Category:
Getting to the top of the search engines and being found by potential customers is easy task. In fact, it takes a lot of time and energy to develop an effective search engine strategy for your business. This is probably the reason that
Category:
There re hundreds of serch engines vilble tody, but some re fr more complex thn others. This rticle will give you n overview of how some of the most populr ones work. Let?s strt with smller engine: InfoSeek. They only index
Category:
Even though most Search Engines have adopted Pay-Per-Inclusion programs, it is still entirely possible to receive plenty of free traffic from the major search services if you follow the rules, and build optimized, high-quality content pages. Actually
Category:
Websites emerging at the top of search engines are almost always able to make money. Search engines offer the advantage of huge traffic for free. Nevertheless for ethical and effective marketing in search engines, it?s necessary for you to use proven
Category:
Site Sponsor
Directory Statistics
Articles: 68252
Categories: 501
Yahoo Entertainment
