Skip to main content

Search Engines Not Indexing Your Internal Pages? Here Is a Trick You Can Do About It

Some bloggers have a problem—only their blog’s home page is indexed in search engines and not any internal pages (or not all internal pages). You may submit a sitemap in Google, Yahoo, and in Live Webmaster Tools for the site’s internal pages to be indexed.

You may wish to check which of your pages are actually indexed by using the ‘site’ operator: Search in Google and Live.com:

site:http://yoursitename.com

This tells you how many pages from your site are actually there in the index of the search engine. If only a few of them are indexed, you have to either manually submit each page or submit a sitemap with all pages (How to Submit a Sitemap for Blogger?)

Google Indexing

Google actually indexes all your pages by following links. It is quite easy to be found by Google, since it normally indexes a page if there is a relevant link coming into it. And as the PageRank increases, the indexing becomes faster. Which means, your newer pages will be found even without incoming links.

For even faster indexing, you can change the crawl rate of the Google bot, by going to:

Google Webmaster Tools->Your Site Profile->Tools->Set Crawl Rate.

If you have the option for faster crawling, you may set it. However, make sure that your web host is robust enough to stand faster crawling. Sometimes, the crawling may cause high data transfer and failure at the hosting side.

Live.com Indexing

With Live, I have a little trouble getting fully indexed. If you check CuteWriting with the site operator, you will see it is not indexed fully in Live.com. I was a bit lax about it and didn’t actually submit my site manually. I believed that with quality incoming links that I have, Live would easily find out CuteWriting and index it fully. But it turned out that was not the case. A few days ago, I found that only 18 of my pages were indexed in Live. Right now, I am in a quest to get CuteWriting fully indexed by them.

Every webmaster trying to get into Live search engine, should have an account in Live Webmaster Tools. It’s inevitable. Also, post your queries to the Live Webmaster Tools forums. You can submit a sitemap to Live, just as with Google. For this, create an XML sitemap and submit it at Live Webmaster Tools.

The Trick to Be Found

Though search engine bots can go deep into a site following links, they miss links all the time. Pages on the home page or one link deeper are likely to be found faster by search bots. Here is a trick you can do for this.

About this WP Premium template here, the archive widget, which has links to all posts is placed in a JavaScript snippet. Search engines do not index anything inside the JavaScript widget. They cannot literally see what it is. So, don’t place any link you wish to have indexed within a JavaScript code. Since my archives widget is placed inside JavaScript on the sidebar, it wasn’t naturally found by Live.

Live search bot has a notorious way of not going so deep into a site. So, it found effectively fewer pages from CuteWriting and thus didn’t index it fully.

So, to get indexed fully, you have to point to the entire posts in the blog from the home page itself or one link deeper. For that, I manually placed each month’s archive pages on the side bar. Not in any JavaScript widget. Just as plain links that search engines can follow. This is exactly like placing a sitemap in normal websites.

Another thing you should be knowing about is the robots.txt text file, which is there in every website (ideally). (Know more about link analysis with robots.txt). In Blogger, the search label pages (the category links of course) are disallowed from search engines with robots.txt file. So, those links will not be indexed normally by Google or Yahoo. I found that Live still doesn’t care (it indexed some of my category pages).

So, don’t place links to category pages. It will not be followed and indexed by search bots. The monthly archive pages on Blogger are not disallowed for any search bot. So, they will be followed and crawled fully. Since each month’s archive page holds all post URLs made in that month, they will be crawled and indexed completely. Look at the bottom of my sidebar, where you can see the links to all months. Live bot now needs to crawl just one link deeper to find all of my posts.

Copyright © Lenin Nair 2008

Comments

  1. The tips you provided to index pages in search engines were really useful.
    I now follow your blog.Your blog is a great resource.
    Thanks

    ReplyDelete
  2. I recently found that Blogger.com is not restricting all the label pages from getting indexed. Monthly archive pages are having no such issues.

    This is such a waste of Link juice as Blogger label pages can play a vital role in improving internal link structure.

    ReplyDelete

Post a Comment

Comments are moderated very strictly

Popular posts from this blog

What Is the Difference Between Hardcover and Paperback?

Today, my reader, Rahman contacted me with a doubt:

Dear Lenin, would you explain why there are two types of books: hardcover and paperback?
This is quite a simple affair and there are explanatory articles to be found at various places on the Net. Here is my addition.

Hardcover

A hardcover aka hardback is a book bound with thick protective cover, with usually a paper or leather dust jacket over the main cover. The aim of hardcover is protection and durability. These books are mainly for long-term use and collectors’ editions. Hardcover books last far longer than the corresponding paperbacks. They do not get damaged easily thus making them perfect for reference guides, great literary works, etc.

In addition, there is a difference in the type of paper used to print hardcover books. The paper used is long-lasting acid-free type. Acid-free paper has a pH value of 7 (neutral) which makes it highly durable. The papers are stitched and glued to the spine.

Hardbacks are prepared for commercial …

En Dash, Em Dash, and Hyphen

We have three types of dashes in use: The hyphen, En Dash, and the Em Dash. In this post, we will see how to use them all correctly.

Hyphen (-)

The hyphen is the minus key in Windows-based keyboards. This is a widely used punctuation mark. Hyphen should not be mistaken for a dash. Dash is different and has different function than a hyphen.

A hyphen is used to separate the words in a compound adjective, verb, or adverb. For instance:

The T-rex has a movement-based vision.
My blog is blogger-powered.
John’s idea was pooh-poohed.


The hyphen can be used generally for all kinds of wordbreaks.

En Dash (–)

En Dash gets its name from its length. It is one ‘N’ long (En is a typographical unit that is almost as wide as 'N'). En Dash is used to express a range of values or a distance:

People of age 55–80 are more prone to hypertension.
Delhi–Sidney flight was late by three hours.


In MS Word, you can put an En Dash either from the menu, clicking Insert->Symbol or by the key-combination, Ctrl + Num…

What Is the Meaning of the Word 'Ghajini'? Story and Trivia of Aamir Khan's New Film [Special]

[Special Entry]



Aamir Khan's latest film is titled a little weirdly for the taste of Hindi filmgoers. 'Ghajini': They have never heard of such a name, and such a word never existed in Hindi or in any other Indian language.

The name Ghajini is the name of the villain of the film. In Tamil version, the name of the villain was Laxman.

As a Tamil moviegoer, I have already watched Ghajini and know the story in full.

So, What Does the Title Mean?

In Tamil, the title of the film is inspired by the story of Mahmud of Ghazni, an ancient invader of India. This person was so persistent in invading India that he continued trying after several failures. In the film too, the protagonist is such persistent in finding out and killing the villain of the film, who had killed his girlfriend, Kalpana (played by Asin). Aamir's Character (named Sanjay Ramaswamy in Tamil), is a short-term amnesiac, who cannot remember anything more than fifteen minutes.

You may ask then how the Ghazni became…