Skip to main content

Search Engines Not Indexing Your Internal Pages? Here Is a Trick You Can Do About It

Some bloggers have a problem—only their blog’s home page is indexed in search engines and not any internal pages (or not all internal pages). You may submit a sitemap in Google, Yahoo, and in Live Webmaster Tools for the site’s internal pages to be indexed.

You may wish to check which of your pages are actually indexed by using the ‘site’ operator: Search in Google and Live.com:

site:http://yoursitename.com

This tells you how many pages from your site are actually there in the index of the search engine. If only a few of them are indexed, you have to either manually submit each page or submit a sitemap with all pages (How to Submit a Sitemap for Blogger?)

Google Indexing

Google actually indexes all your pages by following links. It is quite easy to be found by Google, since it normally indexes a page if there is a relevant link coming into it. And as the PageRank increases, the indexing becomes faster. Which means, your newer pages will be found even without incoming links.

For even faster indexing, you can change the crawl rate of the Google bot, by going to:

Google Webmaster Tools->Your Site Profile->Tools->Set Crawl Rate.

If you have the option for faster crawling, you may set it. However, make sure that your web host is robust enough to stand faster crawling. Sometimes, the crawling may cause high data transfer and failure at the hosting side.

Live.com Indexing

With Live, I have a little trouble getting fully indexed. If you check CuteWriting with the site operator, you will see it is not indexed fully in Live.com. I was a bit lax about it and didn’t actually submit my site manually. I believed that with quality incoming links that I have, Live would easily find out CuteWriting and index it fully. But it turned out that was not the case. A few days ago, I found that only 18 of my pages were indexed in Live. Right now, I am in a quest to get CuteWriting fully indexed by them.

Every webmaster trying to get into Live search engine, should have an account in Live Webmaster Tools. It’s inevitable. Also, post your queries to the Live Webmaster Tools forums. You can submit a sitemap to Live, just as with Google. For this, create an XML sitemap and submit it at Live Webmaster Tools.

The Trick to Be Found

Though search engine bots can go deep into a site following links, they miss links all the time. Pages on the home page or one link deeper are likely to be found faster by search bots. Here is a trick you can do for this.

About this WP Premium template here, the archive widget, which has links to all posts is placed in a JavaScript snippet. Search engines do not index anything inside the JavaScript widget. They cannot literally see what it is. So, don’t place any link you wish to have indexed within a JavaScript code. Since my archives widget is placed inside JavaScript on the sidebar, it wasn’t naturally found by Live.

Live search bot has a notorious way of not going so deep into a site. So, it found effectively fewer pages from CuteWriting and thus didn’t index it fully.

So, to get indexed fully, you have to point to the entire posts in the blog from the home page itself or one link deeper. For that, I manually placed each month’s archive pages on the side bar. Not in any JavaScript widget. Just as plain links that search engines can follow. This is exactly like placing a sitemap in normal websites.

Another thing you should be knowing about is the robots.txt text file, which is there in every website (ideally). (Know more about link analysis with robots.txt). In Blogger, the search label pages (the category links of course) are disallowed from search engines with robots.txt file. So, those links will not be indexed normally by Google or Yahoo. I found that Live still doesn’t care (it indexed some of my category pages).

So, don’t place links to category pages. It will not be followed and indexed by search bots. The monthly archive pages on Blogger are not disallowed for any search bot. So, they will be followed and crawled fully. Since each month’s archive page holds all post URLs made in that month, they will be crawled and indexed completely. Look at the bottom of my sidebar, where you can see the links to all months. Live bot now needs to crawl just one link deeper to find all of my posts.

Copyright © Lenin Nair 2008

Comments

  1. The tips you provided to index pages in search engines were really useful.
    I now follow your blog.Your blog is a great resource.
    Thanks

    ReplyDelete
  2. I recently found that Blogger.com is not restricting all the label pages from getting indexed. Monthly archive pages are having no such issues.

    This is such a waste of Link juice as Blogger label pages can play a vital role in improving internal link structure.

    ReplyDelete

Post a Comment

Comments are moderated very strictly

Popular posts from this blog

En Dash, Em Dash, and Hyphen

We have three types of dashes in use: The hyphen, En Dash, and the Em Dash. In this post, we will see how to use them all correctly. Hyphen (-) The hyphen is the minus key in Windows-based keyboards. This is a widely used punctuation mark. Hyphen should not be mistaken for a dash . Dash is different and has different function than a hyphen. A hyphen is used to separate the words in a compound adjective, verb, or adverb. For instance: The T-rex has a movement-based vision. My blog is blogger-powered. John’s idea was pooh-poohed. The hyphen can be used generally for all kinds of wordbreaks . En Dash (–) En Dash gets its name from its length. It is one ‘N’ long (En is a typographical unit that is almost as wide as 'N'). En Dash is used to express a range of values or a distance: People of age 55–80 are more prone to hypertension. Delhi–Sidney flight was late by three hours. In MS Word, you can put an En Dash either from the menu, clicking Insert->Symbol or by the k

4 Effective Ways to Write About a Boring Topic

  With the plethora of interesting topics to write about, you’re fortunate enough to get the “boring” one. While it can be a pain for many writers to wind up with such a task, I’m telling you now there are ways to make yours more interesting than it is. So if you find yourself stuck with the dreariest topic to fill in a blog about, don’t fret. Here are the four best ways to unburden yourself. 1. Never a boring topic, only a boring writer. Here’s the hard fact: It’s never about the topic being boring. It’s about the writer making it boring. For instance, you’re supposed to write about aquariums. I know, how can you continuously make this topic interesting, right? Well, you’d be surprised just in how many ways you can make it an enticing read. Start by listing down the basic “what”, “where”, “when” and “how” surrounding the topic. You can ask (and research) about “What material was first used to make aquariums?” or even “How the first aquarium was built?” or “What are

How to Remove Duplicate Content From Your Blogger Blog Posts to Avoid SERP Penalty

Duplicate content to an extent may not affect your blog’s search engine rankings. However, there are quite a few times when it can go out of control and start to hit your rankings badly, even without your knowledge. Here are ways to curb it. What Is Duplicate Content Do you have a blog in which you post regularly? Do any two different URLs in that blog have the same content? Then it is duplication. In case of self-hosted blogs, various features like print preview pages, monthly archive pages, category pages, etc., can cause duplicate content . In such cases, normally search engines rank one of the pages lower. However, in extreme cases, when your blog has a number of pages with the same content, the blog can be penalized. Google puts it: In the rare cases in which Google perceives that duplicate content may be shown with intent to manipulate our rankings and deceive our users, we'll also make appropriate adjustments in the indexing and ranking of the sites involved. As a res