Search Engines Not Indexing Your Internal Pages? Here Is a Trick You Can Do About It

Some bloggers have a problem—only their blog’s home page is indexed in search engines and not any internal pages (or not all internal pages). You may submit a sitemap in Google, Yahoo, and in Live Webmaster Tools for the site’s internal pages to be indexed.

You may wish to check which of your pages are actually indexed by using the ‘site’ operator: Search in Google and Live.com:

site:http://yoursitename.com

This tells you how many pages from your site are actually there in the index of the search engine. If only a few of them are indexed, you have to either manually submit each page or submit a sitemap with all pages (How to Submit a Sitemap for Blogger?)

Google Indexing

Google actually indexes all your pages by following links. It is quite easy to be found by Google, since it normally indexes a page if there is a relevant link coming into it. And as the PageRank increases, the indexing becomes faster. Which means, your newer pages will be found even without incoming links.

For even faster indexing, you can change the crawl rate of the Google bot, by going to:

Google Webmaster Tools->Your Site Profile->Tools->Set Crawl Rate.

If you have the option for faster crawling, you may set it. However, make sure that your web host is robust enough to stand faster crawling. Sometimes, the crawling may cause high data transfer and failure at the hosting side.

Live.com Indexing

With Live, I have a little trouble getting fully indexed. If you check CuteWriting with the site operator, you will see it is not indexed fully in Live.com. I was a bit lax about it and didn’t actually submit my site manually. I believed that with quality incoming links that I have, Live would easily find out CuteWriting and index it fully. But it turned out that was not the case. A few days ago, I found that only 18 of my pages were indexed in Live. Right now, I am in a quest to get CuteWriting fully indexed by them.

Every webmaster trying to get into Live search engine, should have an account in Live Webmaster Tools. It’s inevitable. Also, post your queries to the Live Webmaster Tools forums. You can submit a sitemap to Live, just as with Google. For this, create an XML sitemap and submit it at Live Webmaster Tools.

The Trick to Be Found

Though search engine bots can go deep into a site following links, they miss links all the time. Pages on the home page or one link deeper are likely to be found faster by search bots. Here is a trick you can do for this.

About this WP Premium template here, the archive widget, which has links to all posts is placed in a JavaScript snippet. Search engines do not index anything inside the JavaScript widget. They cannot literally see what it is. So, don’t place any link you wish to have indexed within a JavaScript code. Since my archives widget is placed inside JavaScript on the sidebar, it wasn’t naturally found by Live.

Live search bot has a notorious way of not going so deep into a site. So, it found effectively fewer pages from CuteWriting and thus didn’t index it fully.

So, to get indexed fully, you have to point to the entire posts in the blog from the home page itself or one link deeper. For that, I manually placed each month’s archive pages on the side bar. Not in any JavaScript widget. Just as plain links that search engines can follow. This is exactly like placing a sitemap in normal websites.

Another thing you should be knowing about is the robots.txt text file, which is there in every website (ideally). (Know more about link analysis with robots.txt). In Blogger, the search label pages (the category links of course) are disallowed from search engines with robots.txt file. So, those links will not be indexed normally by Google or Yahoo. I found that Live still doesn’t care (it indexed some of my category pages).

So, don’t place links to category pages. It will not be followed and indexed by search bots. The monthly archive pages on Blogger are not disallowed for any search bot. So, they will be followed and crawled fully. Since each month’s archive page holds all post URLs made in that month, they will be crawled and indexed completely. Look at the bottom of my sidebar, where you can see the links to all months. Live bot now needs to crawl just one link deeper to find all of my posts.

Copyright © Lenin Nair 2008

2 Opinions:

  1. The tips you provided to index pages in search engines were really useful.
    I now follow your blog.Your blog is a great resource.
    Thanks

    ReplyDelete
  2. I recently found that Blogger.com is not restricting all the label pages from getting indexed. Monthly archive pages are having no such issues.

    This is such a waste of Link juice as Blogger label pages can play a vital role in improving internal link structure.

    ReplyDelete

Comments are moderated very strictly